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This  report  ms  prepaid  ta  direct  support  of  plsaming  for  th©  Hsoned  I 

Orbital  Laboratory  said  as  a  pest  of  tbs  decussated  effort  of  the  Training  j 

Bessarab  Division  on  the  problems  of  long-tera  2&H1  retention.  ffca  in-iers loo 
planning,  msdiorlag,  enslys®®,  sad  writing  wore  eoes®pli®fc?d  largsly  under 
task  171003*  ted  Factors  in  the  Design  of  Systsas  for  Operator  Training  sad 
^raluatieiu  This  task  is  a  part  of  projest  173.0  s  Essen  Motors  is  the  Usslga 
of  Training  Systems,  ue&sr  th©  direction  of  Sr,  Gordon  A.  Sakstraad*  Tha 
Amda  naedad  for  the  contractual  data  collection,  preliminary  analyses,  end 
reporting  accomplished  by  Kartia^arietta  Corporation  (Baitiser©  Division) 
tssro  drawn  from  resources  available  for  research  on  Busan  Performance  (Budget  j 

Sod®  78OB) »  Most  of  these  funds  -mm  specially  allocated  for  the  purpose  and  | 

the  rest  can©  froa  those  assigned  to  project  6114,  Simulation  Teohniquoe  for  ! 

Aerospaoo  Crews  Trai&iag,  under  the  direction  of  Mr.  Carl  F.  McNulty.  | 

1 

The  actual  test,  which  me  in  two  phases,  ms  condaoted  by  the  Martin-  [ 

Marietta  Corporation  at  its  Baltimore  Division  under  National  Aeronautics  end  ; 

gj&o#  Adainistration  (HA8A)  contracts  K4SiHL034f  Besearch  ca  Pilot  Skill  Baton-  , 

tion  for  Manned  Flight  and  HASw-1319,  Test  of  Pilot  Botssrtioa  of  Simulated  lunar 
Mission  Skills.  Iff*  Milton  A.  Grodsky,  Manager  of  the  Manned  Machine  Engineer¬ 
ing  Dopartseat,  arranged  tbs  details  of  the  test,  the  data,  collection,  ths  1 

preliminary  analyses  and  the  preparation  of  preldsin&xy  reports  (Grodaky  @t  al, 

1964,  1966a).  3s ■  me  supported  in  the  work  by  several  associates,  ©specially, 

J.  A,  Haadeur,  B.  Soborte,  J,  T,  Warfield,  and  T.  M,  Flaherty.  Although  secured 

by  Air  Force  funds,  the  supporting  contract e  (as  indicated  by  their  designation) 

war©  arranged  by  NASA  on  behalf  of  the  Air  Force.  This  ms  vary  dsairoble  V®- 

cm s@  the  work  complemented  directly  and  required  the  us©  of  data,  obtained  under 

the  NASA  contracts  H4Sr*-833  Kid  HAStr-1187,  Baaea  Reliability  Prograa,  also  with 

Martin-Marietta  Corporation,  Baltisscr©  Division,  fir.  Haber  Ho or®,  initiator 

and  technical  monitor  of  the  NASA  contracts  on  hu©sa  reliability,  graciously  ,  | 

attended  to  arxangesaats  for  this  progress  as  well,  participated  in  initial  , 

ssstings  to  got  the  pregrsa  uadwr^ay,  provided  useful  taoimioel  suggestions,  j 

and  was  an  effective  go-betvsen  throughout  th©  course  of  the  effort. 

r< 

Within  th©  Air  Force,  Col  G.  1/utaaa  and  later  (Sol  H.  Allen,  Air  Foroo 
Sjrstsas  Cosaand  liaison  officers  to  NASA,  provided  helpful  suggestions  and 
encouragement  and  sad©  the  necessary  arxengeBsata  with  test  personnel  for  their 
participation,  The  test  personnel  were  12  aerospao®  research  pilots  oho  had 
previously  participated  in  the  human  reliability  progress.  Those  participating 
in  the  first  tost  phase  are  Captains  Jassos  H.  Irwin,  Lachlan  Maeleasy,  Albert  L» 

Atwell,  the  late  Jaaes  3.  McIntyre,  Bobsrt  X.  Parsons,  and  Bussell  J.  Soott  of 

the  U.  3.  Air  FOrc®,  Thosa  participating  in  the  soooad  tost  phaee  are  Captains  (> 

Francis  0.  Bfeuhaok,  Thurlop  H.  I&lph,  Charles  E»  Stone,  and  Janes  Mo  Taylor  of  .s 

the  TJ.  3.  Air  Fores  and  Lieut  scant  a  John  L.  Finley  Kid  Bicksrd  H.  Truly  of  the  J. 

tJ.  S.  Navy.  All  of  the  pilots  responded  to  the  test  rsquireseats  very  well  and 

performed  in  a  thoroughly  professional  Ksamor,  as  ms  Kost  aoosssary  to  tho 

validity  of  th©  test. 
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!?•  Ossfl  W»  t&oss  takings  sal  mgs^sb  ht&pad  ©sssMs®* fcij 

to  asotitato  $&@  tas'gj  &i&?4  tba  Initial  flying;  asis  t!s@  fossil  oszst^aotiM 
assasgoaeats  £os>  tfes  fisest  pMsss  aad  assisted  in  ssaltoriteg  4ata  eoXiesrlisa 
is  the*  lias  .tes  J»  Siisre&y  of  t&s  tj&#sss&#  ©f  Bsyfaia,  iss*fe&sa& 

tss&ss?  ecsrtxmct  IF  33(615  )-X£&4$  Fa#t^Xoigi«aX  ®$8*asi*S»  «  Ogk^&  fzMM&gp 
psatixmd  diligently  as&  gsee^la3n&g3y  ty  Mad  m  ess^.;g&&£3  Of'tbf'^oaXj^ 
iio&s  2#©d2®d  for  tbs  asia  as&lf&iac  H&»  loMsrfe  S«  $®WS  sSM  @t  tie-lMtsssli 
of  Beytos  mi  mxfctesz  ttado?  tbs  ass®  os&irasif  eoogs&lia&M  Easy  of  i&§  cgsciaX 
Sat®ss*®tiv»  easlye«c  and  etm’sd  as  ®  stisalstiag  dissaiseat  of  th&eaaljrtio 

‘i’"  lawol'sa^* 


Pour  ©re«3  of  t hs$*  aasoj^ao*  ressar-ch  pilots  «s?3  t©gt$&  oa  ©  si^sjl&t&d 
?-*d5y  lunar  landing  mission  s§  different  intervals*  epprssasg&lBg  4*  8,  9,  caad 
13  m$&as  after  original  training,  fSls  6  m^s  &t  trains^  had  ca3®2as$M  1» 
tfc®  real  tiae  pesfomms^  of  the  aissica  test,  for  the  skill  raiesfeisa  test  tfe© 
aissisa  was  eo^ jessed  into  a  siagl®  13-hejs?  workday  by  csdasioa  of  less 
eigs±ti&mh  tasks  sad  waiting  periods,  fallowing  the  test  one  or  thrss  days 
of  additional  tmiaiag  oa  selected  sieaion  phases  v&a  given  all  eresn, 

ThM  analys  is  of  results  focused  attest  ion  os  iadividaal  m&  crew  parfbrsar 
a bo©  at  the  ©ad  of  training*  in  the  skill  retention  teat  aiesica*  and  in  the 
following  retraining  trials,  as  represented  by  22  ©eleoted  flight  control 
parameters  distributed  ovssr  nim  mission  phases,  %  the  use  of  novel  amlyfci# 
techniques  the  levels  of  performance  observed  were  represented  as  rsliaMlitiea* 
or  probabilities  of  euooass  in  nosiing  bypothatissl  criteria  for  the  pasesa@t@m 
disc,  for  greater  sensitivity  to  changes  in  capability,  test  and  rstraiai^j 
psrforsanoss  were  alternatively  represented  as  probabilities  of  success  la 
Eseting  th®  level  of  perforaaace  eatinatad  achievable  by  each  individual  in  95$ 
of  his  performances  at  th©  end  of  training. 

On  careful  ©valuation  as  to  possible  biases  the  obtained  probabilities 
are  ta&aa  to  indieat©  (l)  that  lack  of  diroot  practice  of  critical  tasks  eves? 

8  rpseka  or  twro  in  long  duration  ©pace  ©lesions  will  result  in  uascosptalde 
skill  deterioration  unless  suitable  readies  are  sought  in  design  and  op er&» 
t local  plesaingj  and  (2)  that  aorospaco  research  pilots  are  capable  of  perform¬ 
ing  the  type  of  ©lesion  used  in  this  study*  providing  extras®  care  is  given  to 
their  training  and  their  individual  performance  reliability  is  dss>onstratod. 
Weeds  for  farther  research  on  skill  retention  are  indicated  and  the  advantages 
of  the  novel  analytic  methodology  used  are  stated. 


mm  cp  camsm 


Xe  smosre?!©? 

Purpose  of  the  Study  *  «  •  • 

1st  sat  sad  Mediation  of  this  Bspost 


XX* 


fH3  HiffOSS  OP  SBB  TEST 

SlO  T$&t  f®C8(XBB&  1  • 

Tits  Sissslsted  Mission  sad  Previous  Training  . 
feat  Performance  E©^air$sssBts  »  «  <  «  * 

Instrumentation  •••«••»« 
Test  Procsdnrss  ........ 

Ps^orgaac®  llossures  •  «.«•<>• 

Analytic  Methodology  »  •  •  •  »  .  • 


4 


a 

* 

4 

o 

• 

• 

$ 

c 

• 

9 

♦ 

• 

• 

B 

• 

• 

23 

• 

♦ 

» 

24 

0 

w 

* 

27 

m.  mam 

Skill  hsml  Attained  la  Training 
Esteation  Tost  Mission  Psrfo2®aac© 
Eotmiaiag  Perforassaos  .  •  • 


32 

32 

38 

48 


xv.  TrnmmmKM 

Data  Collection  Procedures  .  • 

Analytic  Proosteeo  •  •  • 

Th&  Problem  of  General  iasbil  ity 
Th©  Considsxod  Findings  •  • 


91 


v.  imxe&Txess 

8am®  SgBfem  Bssign  ee@s,eS*.eco 

Methodology#  Tho  Ksasureaant/Pre diction  of  Siuan  Perforsanos 
Further  loseareh  •  •••.  ••••<>• 


95 

CR 

99 

102 


VX.  SunMABT  104 

AFPiraim 

1,  Sasnaxy  of  Snitching  Bata 
n,  Botes  on  Analytic  Methodology 
IH.  Probabilities  for  Each  Measured  Parojastor 
XV.  Graphs  of  Bets  froa  F-131 

Tjwmmffisa 


« 


I. 

II, 

in, 

IV. 

v. 

VI. 

vn. 

vm. 

So 

s. 

sc. 

m. 

mi. 

XIV. 

XV. 

XVI. 

xvn. 

XVIII. 

XIX. 


f 

1 

l 


list  e?  i 

t 

i 

l 

i 


Bata  on  the  'Jest  Persocnsl  5 

Schedule  for  0-4  a&d  0-9  Seat  Mission  10 

Schedule  for  0-8  and  0**X3  Tost  Mission  11 

Repeated  Trials  on  Mission  Phases  12 

Measures  Used  in  Analysis  of  Flight  Control  29 

Structure  of  the  Bata  30 

Reliability  Per  Criteria  at  the  Sid  of  Training  35 

Reliability  Per  Criteria  in  the  Test  Mission  39 

Change  is  Mission  Reliability  Per  Criteria  froa  Training 

to  Test  Mission  42 

Probability  Estimated  fros  Test  Mission  Performance  of 
Attaining  tbs  Pa^  Skill  Level  Achieved  in  Training  45 

Loss  in  Test  Mission  Reliability  fTesa  Training  Lsm>l  47 

Reliability  Per  Criteria  in  Early  Retraining  50 

Reliability  of  C~8  and  0-13  Per  Criteria  as  Estimated  fros 

Boat  Four-Trial  Hook  in  EstmLsing  51 

Oosaparative  Reliabilities  of  Cr&rs  0-8  and  0-13  in  Trainings 
Test  and  Retraining  52 

Comparative  Reliabilities  of  Creisg  0-4  and  0-9  in  Training, 

Test  and  Retraining  53 

Probability  Estimated  from  Early  Retraining  Porfoasanee  of 


Attaining  the  Skill  Level  of  the  Last  Four  Training  Trials  56 

Probability  latimted  frosa  the  Bast  Four-Trial  Block  in 
Retraining  of  0-8  and  0-13  Attaining  the  p  qc  Skill  Level  of 
the  last  Four  Training  Trials  *  57 

Mission  Reliability  of  0-8  ©ad  0-13  Per  Criteria  in  Training 
as  Entiaatod  from  the  Last  Four-  end  the  Rost  Four-Trial  Hook  68 

Change  in  Mission  Reliability  of  0-8  and  0-13  froa  Training 
to  3bst  Four-Trial  Hook  in  Retraining  71 


vi 


5 


mm  0?  mm s 

(Goniiaaad) 


M 

XX. 

Cfeaage  in  Mission  Esliability  of  0-8  sad  C-33  frca  Sssiaisg 
to  Boat  Four-Trial  Hook  in  BstraisiBg  p®r  Alternate  Safer- 
sme-e  Measures 

n 

ro. 

Perfczeano©  of  P-131  in  Tr&itdagg  in  ik®  Tssfc  Mission  end 
in  the  Bast  Stour-Trial  Block  in  Hotminiag  Per  Sosaal  and 

J,  H^ributicms 

74 

mx. 

Eslatioaship  bstwson  Order  of  Parsaster  Perfcsrsanc©  ia 

Trsini sg  and  Bsst  Four-Trial  Hook  in  Ect  mining 
(XendaH*s  r) 

80 

xmx. 

Correlations  of  Parasatera  within  Phases  on  the  Last  Pear 
draining  Trials  (Xandall’s  'T ) 

82 

m 

Skill  Bsieafcion  Teat  Mission  S^itchii^  Sr-rora  by  Phase  and 
Workspace  (in  appendix  I) 

108 

m. 

Bali&bility  Par  Criteria  at  the  led  of  Training 
(in  appendix  HI) 

118 

XXVI. 

Eoliability  Per  Criteria  in  the  Test  Mission 
(in  appendix  III) 

119 

mil. 

Probability  Estimated  frea  Test  Mission  Performance  of 
Attaining  th®  p  ok  Ls^©1  of  the  Last  Pour  Training 

Trials  (in  appendix  Hi) 

120 

ran. 

Eeliability  Per  Criteria  in  larly  Eetmining 
(in  eppendix  HI) 

121 

XXIX. 

Probability  Estimated  fros  Early  Bstraining  Perforsane© 
of  Attaining  the  p#^  Skill  Lsval  of  th©  last  Pour 

Training  Trials  (in  appendix  Hi) 

122 

XXX© 

Eeliability  of  0-8  and  0-13  m  Estimated  froa  Esot 

Pour-Trial  Hook  in  Eat  raining  (in  appendix  IH) 

123 

vil 


ram  o?  mmm 


1.  Right  Rear  Yi&s  of  Cosssand  Model©  13 

2,  ffconi  Yi©»  of  Slsalation  Hgpipssat  15 

3»  Main  Ocsasam  Module  Pa&ale  16 

4c  E&vigator'o  Station  It 

5®  Bear  View  of  Sisalstion  Sqaipsant  20 

6®  Simulation  Costal  Eooa  21 

7.  Phase  Reliabilities  of  Ccabiasl  Srsraa  at  the  Rfed  of  draining  34 

8«  Phase  Beliabilif ioa  of  P-8l  aid  P-131  in  draining,  in  the 

Test  Mission,  and  in  their  3s st  Foor-^rial  Block  of  Entraining  41 

9c  Szpecrfced  and  Obtained  Phase  Prohahilities  of  P—81  and  P-131 
Achieving  their  p  as  Levels  of  Training  in  the  feat  Missies 
and  in  thoir  Best  Fmrc-Trial  Hook  of  Set  raining  $6 

10.  Hypothetical  Mstribation  at  the  Sad  of  draining  of  P-231ss 


Vertical  Bate  at  Ispact  on  Imssr  leading  indicating  Probability 
of  Mooting  the  Criterion 


1X2 


♦ 


R&mTTIOI?  OF  SIMULATED  LUNAS  LANBSHQ  MISSION  SKILLS t 
A  ®  OF  PILOT  H2LIAB1LXTY 

SECTION  I 
INTHOOTCTION 


Tho  requirement  to  perform  speoifio  duties  and  tasks  after  extended 
periods  daring  which  they  are  not  performed  is  a  familiar  experience  in  both 
military  operations  and  everyday  living~snd?  incomplete  recall  under  thee© 
circumstances  is  a  familiar  sequel*  However,  the  need  for  long  retention  of 
critical  knowledges  and  skills  has  seldom  been  so  dramatically  emphasised,  or 
become  such  a  matter  of  importance,  as  it  has  in  relation  to  planning  for 
extended  space  operations.  For,  it  seems  evident  that  in  the  space  context 
there  will  b©  in  the  foreseeable  future  circumstances  in  which  there  is  limited 
or  no  opportunity  to  engage  in  practice  operations  for  training.  Furthermore, 
there  is  a  significant  difference  between  routine  activities  and  space  opera¬ 
tions  that  gives  additional  reason  for  interest  in  retention. 

In  everyday  living  and  in  routine  military  operations,  there  is  often  a 
considerable  tolerance  for  error  so  that  the  quality  of  performance  may  vary 
widely  and  still  be  counted  acceptable.  But,  in  current  space  systems  (as  in 
the  newest  high  performance  weapon  systems),  the  initial  and  operating  invest¬ 
ments  are  so  large,  the  criticality  of  the  missions  so  great,  and  the  risks  of 
failure  so  numerous  and  severe  as  to  seriously  call  into  question  traditional 
tolerances  for  human  error.  In  this  context,  then,  of  more  stringent  human 
performance  requirements  in  increasingly  lengthy  missions,  perhaps  not  inher¬ 
ently  affording  opportunities  for  practice  of  critical  tasks,  it  is  important 
to  ask,  Would  operator  performance  be  sufficiently  degraded  through  the  process 
of  normal  forgetting  as  to  require  soma  kind  of  special  remedial  attention? 

Of  course,  what  one  is  really  interested  in,  is  the  more  specific  ques¬ 
tion,  Would  the  forgetting  of  critical  skills  on  the  part  of  this  (or  these) 
particular  operators)  in  this  particular  system  performing  this  particular 
mission  so  adversely  affect  the  probability  of  mission  success  as  to  warrant 
special  attention  in  design?  The  question  itself  clearly  implies  that  oertain 
kinds  of  information  must  be  available  if  an  accurate  answer  is  to  be  given. 
First,  it  is  necessary  to  know  what  the  relationship  is  between  mission  tasks 
performed  by  the  operator  and  the  probability  of  mission  success.  Those  tasks 
which  are  most  relevant  to  mission  success  and  which  carry  stringent  performance 
requirements  are  naturally  of  primary  interest.  Second,  it  is  necessary  to 
know  the  nature  of  those  critical  tasks  in  sufficient  detail  so  that  accurate 
estimates  about  their  retention  can  be  derived.  Sons  kinds  of  asks  are  more 
quickly  forgotten  than  others.  Finally,  it  is  nacessary  to  know  the  capabil¬ 
ities  of  the  operators— how  well  'they  can  acquire  and  retain  such  skills.  Even 
very  competent  test  pilots  differ  in  their  skills  at  the  various  kinds  of  . 
activities  whioh  must  bs  performed  on  a  given  mission  and  in  overall  capability. 
Furthermore,  the  competent  individual  seldom  is  capable  of  performing  all 
aspects  of  a  mission  equally  well. 
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If  this  Is  iha  fcSM  of  isfbssssfci m  for  ks  aecas&to  to  tbs 

qassttoa  of  skill  retostlssa  to®a  it  is  &gggz%2&s®m  t®  ask,  lbs?  css  d&taited 
tofcrsatica  m  task  criticality,  epewlfka  eslftieal  tasks,  eM  ep^srata?  skill 
acquisition  end  retention  's§  ©Mateos!?  !maa!^t  £s£&ss££$1ssb  es  tbs  ealt- 
iealiiy  and  astose  of  taste  co«ld  bo  ava&XffiS&©  aa  <§®aigp*  &&&«  2tet,  to  ebtsia 
tofcxsatiesa  ea  c*.  tries’  skill  asd  skill  retest  ton  obviously  mma  to 

royals®  o  tesft  la  tins  actual  sisuissi.  This,  of  ©c&rea,  is  salad  sat  as  ig^rs®®- 
tical.  Shat,  tosea,  are  tbs  alt®23aatiwa  tc  sash  @2  eparsticssal  test? 


Naturally,  0 aa  thinks  of  aimiffitlon  tsete&sm  os  aesaa  of  presiding 
realistic  operations  for  test,  dssiga,  end  training  parposse.  To  obtain  the 
nos&ed  tofersattca  mi  pilot  skill  reteatica  tbay  sight  te  ui§©&  to  aitonr  of 
t-so  ways.  *  (hs®  ssihod  ooold  provide  realisa  tores^i  tbs  ©psrstlcn  of  esa  ox 
acre  cpeotolly  mdlftod  flight  Stoles  so  that  at  least  tbs  pcrfeomaee  of 
certain,  presssably  critical,  atoaica  eagpasais  could  la  otodtod.  Sxa  ether 
would  provide  real  tea  thresgb  the  operation  of  a  adsatoa  BiEalc&ar.  Bit  toare 
are  specific  disadmatagsa  to  to®  ua@  of  ©itkar  tscksiqas  to  tost  toa  stealsn 
context  cannot  ba  effectively  provided  with  toas.  Tbs  ssophistiost^d  ssieesisa 
simlator  ®y  todasd  provida  too  toll  xuag©  of  aicsioa  tasks  properly  totore 
looed,  shared  sad  s®q$ss©ed  to  tin®  with  those  tasks  of  epeoial  interest  sad  a 
aiasicea  of  toll  fiaraticra.  Ecssmsr,  it  does  so  at  too  ©s^es&o  of  m  realign 
to  took  coos  mUr,  mm  toportesctly,  realistic  b&E&rda  of  too  aofeal  opsratisa. 

On  too  otter  band,  spsotol  training  vehicles  presently  flail  far  short  of  prcvld- 
tog  tbs  toll  rang©  of  tasks  ozonised  eM  tessporally  related  to  a  realistic 
faahion  to  ateoicna  of  scything  near  realistic  dare&toas*  Also,  ovraa  the 
flight  risks  are  not  ocEparaldy  great,  Thus,  tbs  interpretation  of  any  toot 
remits  obtained  ty  toes©  asana  east  compensate  cosetew  for  tbs  lack  of  reslim, 

Thera  are  still  other  difficulties  involved  to  gaining  information  usatol 
for  deciding  absast  tbs  skill  retontica  pryblea  if  this  is  to  bo  dsn s  very  early 
to  toe  pro ccssj  of  aystoa  darolopcssait.  At  an  early  stage  tbs  lama  taste  have 
act  jet  bw»  olssrly  idsstiftod  ssd  defined  and  their  oriti«®lity  to  ret  teosa. 
Portoszsaor©,  toe  operational  paromsaol  for  apaeifio  oixaioaa  generally  tssm  not 
boon  identified.  Aa  a  result,  it  to  ispossible  to  obtain  the  relatively  precise 
information  moded  cm  toe  proper  pcrmamol  perforetog  relevnat  taste  or  to 
interpret  too  exact  significance  of  emy  skill  lossea  observed.  And,  ocnslssicns 
drawn  tberefroa  booesao  generally  ©eater  end  less  ttsotol  for  dcoiaicna. 

In  view  of  snob  difficulties  in  aseesabltog  snffioieat  inforssatiesi  fma 
direct  tests  for  early  decisions,  it  coy  tbaa  be  soppoeed  tost  the  problem  ossx 
bo  ocnveniently  handled  on  the  basis  of  previous  espsriaental  studies  of  skill 
retention.  There  bare  boon  a  neater  of  experismts  oondaoted  since  1900  on 
this  topic  (Saylor  end  Brings,  i960),  but  few  uesful  principles  for  design  bare 
been  or  oeai  be  extracted  toon  that  research.  The  coin  reasons  are  that  stoplo 
tasks  hardly  representative  of  operational  taste  hare  been  used,  the  subjects 
ueod  to  these  studies  hare  not  been  typical  of  individuals  too  will  bs  called 
upon  to  porfore  space  atosions,  end  toe  validity  of  cany  studies  is  doubtful 
boccaso  of  unsolved  cethodolGgical  prcblcsxs  in  oonduoting  then.  Thus,  frea 
this  available  information  it  cay  bo  concluded  sorely  that  oooa  dooreocat  cay 
occur  as  a  function  of  (l)  toe  nature  of  too  task,  (2)  the  degree  of  skill 
acquired,  (3)  the  retention  interval,  end  (4)  tbs  obarootoristioa  of  the  pore 
forcer.  Little  elec  esy  bo  assorted. 
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f Ms  study  ms  cm  of  e®mm 1  sssosst  attests  ty  to  Eshsvlcrsl  Istes 
fctomiery  to  gain  bettor  -lata  on  to  ©igaituda  of  to  skill  nstes&isa  gzoULm, 
in  spaea  operations.  As  gush,  it  ©ay  fc©  placed  ca  to  sdMie  grossad  eesswhsr® 
baftossa  to  last  alternative  jusrfc  azblinoio  For,  &  dsfinit©  effort  wan  sada 
to  sms  sans  representative  data  by  having  teat  psffsossasig  representative  of 
those  to  will  perfsrs  sgaoa  edasloss,  p$r£era  tesla  Ito  toe®  to  he  emcoisxtsrod 
in  epaco.  Booms©  cf  this  to  resalts  sns  s®r®  dirasfcly  applicable  ton  soot 
svail&fel©  data  in  to  litemtoa  on  to  topic.  Gto  to  otfcar  hand,  because  so 
rssl  ssyataa  woo  involved  In  to  tost,  emot  Information  on  task  criticality  ms 
looking  and  absolute  elgnificaaoa  for  sissim  sooaess  csnaot  he  dstorsinad, 
*?hsr®fen®,  to  results  sstafc  Is  cosai&axed  relative  to  to  lisdtatioas  of  to 
siKolat ion  used  e®d  to  prior  skill  level©  attained  by  to  tot  possess®!,  Sbia 
say  not  ho  as  serious  a  drawback  as  night  ho  eappoasd*  Eolaiiwo  inf  oration 
my  h®  sore  useful  aayhm?  because  there  is  no  easarsac®  that  specific  toko, 
task  criticalities ,  operating  esviniasasta,  oad  skill  l«vsla  will  run  parallel 
in  different  mtstsss, 

More  specifically,  the  intent  of  to  study  was  to  obtain  a  batter  ©atisate 
than  ms  otherwise  available  of  the  dsgre©  of  skill  loos  in  typical  spao®  vehiol® 
operations  took  eay  bo  exacted  over  periods  up  to  3  eoato  duration,  For  this 
purpose  a  relatively  oosploto  simulated  space  mission  ms  used,  m  performed  by 
specially  traimd  aorospace  rsssarch  pilots; 


Preliminary  reports  of  the  study  (conducted  in  to  phases)  lm  boon 
pr@p3Z9U  He  An  Orodaky,;  J.  A*  Kandour,  «t  si  (1964),  end  by  Mu.  A.  Grodsiy, 

D.  Eobarts  end  J.  Msndour  ( 1966a),  to  wore  directly  responsible  for  its  eameur 
tiers.  Kosovar,  a  different,  coro  intensive  analysis  cf  to  data  than  is  pre~ 

® anted  in  tbcco  reports  semed  advssrtagoena  by  my  of  providing  for  a  core 
precis®  statement  of  results  and  additional  interpretations.  Also,  m  a  oonr 
vsnienc®  to  uaora  of  to  information,  a  eiaglo  dessriptian  of  to  study  eccsod 
desirable.  Accordingly,  this  report  is  intended  as  a  relatively  cospleio 
description  of  to  study  with  eerpbaoia  upon  analytic  results  end  toir  implica¬ 
tions.  Some  details  on  tasks,  instrumentation,  and  test  proesdusoa  have  boon 
oaittod  and  for  tos®  to  reader  tsay  refer  to  tbs  preliminary  reports  and  to 
olosaly  related  report  by  Qro daisy,  et  al  (1966b). 
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la  brief,  tbs  ta<rfc  required  tlx®  perfossssc®  of  a  simulated  lunar  landing: 
mission  and*  additionally ,  eoleeied  tasks  fswa  that  aiasica  try  four  r@e«ar®b 
pilot  crows  that  had  "been  trained  various  psi-iods  of  tigs  earlier  in  th©  laiasion 
tasks.  Tha  simulated  test  aission  was  performed  is  coaprassed  tins  so  that 
intervals  requiring  no  critical  mission  activities  wars  greatly  shortened. 

Also,  each  oretesaabar  performed  as  oretr  oo wmH&s  in  each  min  p Mm  so  that 
measures  mm  obtained  for  each  individual  of  each  crew.  Multiple  measures 
of  performance  ware  obtainad  for  each  pirns®  (with  a  tm  exceptions)  and  th© 
ecvarel  kinds  of  task  parssoters  involved  ©ad  related  to  the  previously  observed 
performance.  In  this  osetiea  th©  details  of  the  test  concerning  the  test  per- 
stamsl  and  thair  previous  training,  th©  test  performance  requlrssisats,  the 
instrumentation,  and  th®  procedures  used  for  testing  mid  data  analysis  arc 
briefly  presented. 


The  test  personnel  consisted  of  12  aerospace  research  pilots  who  mm 
ell  graduates  of  th®  Asrospaoe  Ra search  Pilots  School  at  Edwards  Air  Force  Ease. 
At  the  tine  of  teat  10  had  ths  rank  of  Captain  mid  two  were  U.  S»  Eavy  Lieuten¬ 
ants.  They  mm  currently  assigned  to  eaperissnt&l  or  instructional  flying 
positions  with  the  Mr  Fore®,  end  have  bachelors  degrees  in  either  engineering, 
physical  science  or  ailitaxy  science.  Their  background  includes  considerable 
military  sad  flying  expsrisnos,  especially  in  fighter  aircraft,  ©a  shorn  by  the 
listing  in  table  I. 

Before  testing,  tha  12  pilots  had  participated,  as  crews  of  three,  in 
extensive  training  in  the  simulated  mission.  This  training  ms  acocspliehod 
as  a  necessary  part  of  tha  national  Aeronautics  end  Specs  Administration  study 
of  bussan  reliability  dsaoribsd  in  th®  report  by  Grodsky,  ©t  al  (1966b),  Two 
of  tho  crows  had  trained  consecutively  in  ths  lato  spring  and  sussaor  of  1964 
and  th®  other  two  had  trained  consecutively  in  the  sussssr  and  fall  of  1965* 

This  oircusatano©  mda  it  possible  to  obtain  a  natural  variation  in  retention 
latarml  by  tha  simple  expedient  of  ocnsooutively  testing  each  pair  of  orevs 
later.  Tima,  th©  first  tiro  crave  returned  at  such  a  tins  as  to  provide  rot  action 
intervals  of  4  end  9  weeks  and  the  go c end  two  orotra  returned  at  euch  a  tine  as 
to  provide  retention  intervals  approximating  8  and  13  weeks.  Accordingly,  tho 
cross  are  referred  to,  for  oonvenienc®,  as  0-4,  0-9,  0-8  and  0-13.  Similarly, 
individual  pilots  (?)  of  the  crews  or®  designated  by  a  number  (l,  2,  or  3) 
indicating  th®  order  in  which  they  performed  in  test.  Thus,  F-91  is  the  first 
plLot  tested  in  the  9“W9®k  crcv  end  he  was  followed  by  P-92  and  P-93. 
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Tba  sibilated  Elesisa  for  which  tk©  ores©  bad  pr^croaly  rsoaived  train¬ 
ing  ms  a  7-day  lunar  landing  eiesiea.  Tha  Miaulatiea  began  with  an  escent 
(eutomtisally  coat  rolled  to  100,000  fast)  tc  ©  100  EH  psrsing  orbit  md  c<m- 
timed  through  the  following  succession  of  sajor  activities? 

trsnsXua&r  insertion 

transposition  of  flight  sodalos 

position  detszKioationa  sad  nidcoars©  corrsationn 

lunar  orbit  insertion 

lunar  landing 

lunar  ©scant 

tnaasssrtk  insertion 

position  dat erolnat ions  md  e&docurse  con's ot ions 
earth  entry 


For  ssaguresssrt  purposes  it  teminatsd  sdth  the  deplt^ssent  of  the  drogue  shat© 
at  an  altitude  of  25,000  feet..  Interspersed  appropriately  smug  these  etijor 
activities  were  a  nuabor  of  secondary  chaoknut  md  preparatory  activities  as 
■soil  as  sisiulatQd  ©ssxgsaoies. 

Most  of  the  simlsted  flight  ms  seoaspliahed  ty  tha  crew  of  threo  frees 
within  a  cossasad  nodule.  CroRSsabers  m  noainally  designated  ooszsandor,? 
navigator,  and  engineer,  hut  each  meaber  had  his  turn  at  perfuming  each 
important  activity,  Ths  nisaioa  plan  ms  arranged  so  as  to  provide  normally 
on-duty  periods  of  2  homo,  off-duty  periods  of  2  hours,  end  two  4~h#usr  sl®sp 
periods  ©very  24  to  26  hours.  Sleep  periods  were  generally  preceded  md  followed 
by  off-duty  periods.  Exceptions  to  this  scheduling  occurred  in  the  lunar  lauding, 
exploration,  and  takeoff  phases,  which  wore  accomplished  ly  two  of  the  three 
crerasssahers  frost  aboard  a  separate  excursion  nodule.  For  these  phases,  crow* 
Ev$s>bar*  «aw  on  continuous  duty  froa  8  to  14  hours,  dspeadiag  m  position.  Tm 
result  ms  that  o vsr  the  couplets  aisaion  each  eressssaber  spent  apprcodEatoly 
70  hours  on-duty,  50  feeare  off-duty,  end  50  hours  sleeping. 

*o©  oritioal  flight  activities  which  had  to 
be  accomplished  for  a  successful  aisaion  are  briefly  charoetarised  as  follows? 

1.  Tranalunar  insertion.  After  about  2  hours  in  parking  orbit,  a  regain¬ 
ing  boost  propulsion  unit  is  relit  to  obtain  the  additional  velocity  ( A?) 
required  to  aeoape  earth  orbit  and  enter  a  troaatunar  trajectory.  This  requires 
interrogation  of  a  guidance  eoaputer  for  inforeation  {on  vehicle  attitude,  initia¬ 
tion  tin®,  and  AV),  proper  attitude  control,  and  ticoly  initiation  and  cutoff 

of  propulsion  in  aoeordanc®  with  the  inforc&ticn  provided. 

2.  Transposition.  Beginning  with  an  initial  configuration  oonaisting 
of  first  the  command  nodule  {(Si),  then  a  service  nodule  (Si)  containing  flight 
propulsion,  then  the  lunar  excursion  nodule  (l O),  and  a  regaining  boost  pro¬ 
pulsion  unit,  a  rearrsngesoat  of  nodules  is  aoco^p  licked.  This  requires  first 
initiating  an  autoaatio  separation  sequencer  that  separates  the  S3  frees  the 
LEI,  jettisons  tho  IO  adaptor,  and  accelerates  the  (2J-SM  oenbinatica  forward, 
and  stops  it.  Than  tho  pilot  pitches  the  (23-SI  180°,  flies  it  backward  with 
reference  to  a  jsoter  display,  and  docks  it  against  the  LSI  upper  hatch.  The 
regaining  boost  propulsion  unit  is  then  jettisoned. 
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3o  Position  determinations  end  tsidoourse  corractioaa*  Position  detesBina?- 
tions  and  say  necessary  si&sourg®  corrections  era  sad©  at  four  times  daring  lath 
the  translunar  end  the  traasosrth  coast  phases*  Those  involve  making  optical 
sightings  with  a  navigational  instrosont  to  seasurs  the  angle  between  an  eerth 
or  tsoon  landmark  and  a  star,  as  wall  as  star  sightings  to  obtain  data  for  th© 
guidance  computer,  Using  the  automatically  calculated  values,  the  inertial 
measuring  unit  is  then  aligned*  On  the  basis  of  the  corrected  navigational 
inf  creation,  as  analysed  by  the  computer,  the  pilot  then  mtesa  the  indicated 
velocity  increments  is  the  appropriate  direction  as  to  modify  the  trajectory 
toward  the  desirod  path, 

4*  Lunar  orbit  insertion.  On  the  far  side  of  the  noon,  at  pericynthia a, 
retrothrust  is  aeamally  applied  to  reduce  the  velocity  appropriately  to  achieve 
an  80  EH  circular  lunar  orbit*  Th®  ©HU  combination  regains  in  this  orbit  while 
th©  lunar  excursion  is  carried  out® 

5*  Lunar  landing,  Th®  lunar  landing  sequence  is  composed  of  several 
distinguishable  subsequences*  Operated  by  two  craussabars  (ihila  the  third 
remains  in  th©  ©5),  the  IM  is  first  separated  frca  the  £H,  translated  clear  of 
,it  and  stabilized  in  tbs  o&as  circular  orbit.  Heart,  at  104°  central  eagle  frca 
the  landing  site  with  th©  LSI  reoriented  toward  the  center  of  the  mm,  dssomt 
propulsion  is  fired  at  the  proper  tins  to  achieve  am  elliptical  orbit  with  a 
periluns  50,000  fest  over  the  loading  site  without  change  in  orbital  period® 

A  ballistic  descant  trajectory  is  than  followed  with  attitude  being  controlled 
to  permit  radar  t rooking  of  the  £21*  At  the  proper  tiaa,  braking  is  oarrlsd  out 
by  orienting  the  IM  and  filing  the  lending  engine  to  provide  a  thrust  vector— 
the  crow's  line  of  sight  bsiag  90°  relative  to  both  orientation  end  vaster  and 
toward  the  lunar  surface.  Pitch  and  roll  adjustments  arc  used  to  control  alti¬ 
tude  and  lateral  displaeaaaat  and  thrust  level  is  varied  to  achieve  zero  vdcoity 
directly  over  th©  landing  site*  The  vehicle  is  then  pitched  up  to  noxml  attitude 
and  thrust  adjusted  for  a  hovering  position  at  1000-2000  feat,  Fin&tly,  thrust 
is  reduced  to  aohiove  an  acceptably  lor  rat©  of  descent  to  the  landing  Bite  with 
translational  corrections  sads  to  avoid  displacement  frca  th®  exact  touchdown 
site  desired* 

6*  Lunar  ascent*  Upon  coapletica  of  exploration  and  other  duties  on  th© 
lunar  surface,  th®  IM  anew  prepares  for  ascent  by  separating  th®  landing  engine, 
abandoning  it  at  th©  site.  The  ascent  engine  is  fired  with  the  vehicle  oriented 
for  a  vertical  trajectory,  yaw  being  introduced  during  initial  ascent  to  obtain 
downraage  views.  Pitch  is  then  introduced  to  achieve  a  desired  ascent  profile 
to  50,000  foot  and  velocity  is  adjusted  to  achieve  a  Eobssn  transfer  orbit  with 
apiluna  at  the  ®  orbital  altitude*  The  sequence  is  initiated  at  a  tine  which 
will  result  in  th©  IM  being  ahead  of  the  (Si  in  orbit*  Pitch  end  roll  adjust¬ 
ments  are  used  to  control  altitude  end  out-of-plane  notions*  Heart,  with  the 
engine  off  and  the  vehicle  coasting  around  the  noon  gradually  gaining  altitude, 
the  vehicle  is  oriented  so  that  the  CK  can  be  acquired  end  tracked  by  radar 
and  seen  (at  about  20  O  separation)  by  mens  of  flashing  beacon*  Bcmdosvcas 
in  then  aocoaplichcd  by  using  translational  thrust  to  adjust  the  orbital  velocity 
vector  to  natch  that  of  the  ©£,  less  a  dosing  range  rate.  This  rat©  is  gradually 
reduced  for  stabilisation  in  the  eosa  orbit  with  a  mail  separation  distenoo. 
Finally,  docking  is  accomplished  by  controlling  the  IO  attitude  end  traaalar* 
tic®  to  obtain  dosing  at  a  very  low  rate  with  the  IM  forward  hatch  end  the  ©I 
forward  hatch  aligned*  Upon  transfer  of  tb®  lunar  excursion  teen  to  the  C&, 
the  IO  is  left  in  lunar  orbit* 
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7*  fssaseas'th  osfhit  temstics i*  Sufficient  ®8$teigml  vaXeaiiy  is 
obtained  %  fiySjRg  the  ©srvies  to  escspo  th»  lt»  orMt  ©ad  ester  a 

tg®sg&&rth  Afctiisd®  ©safarcCl  sad  iisely  ialtiatiaa  m&  @s&&££  of 

tbs  sagls®  fecoerdiag  to  data  p®e??idsd  th@  gsids&s'-s  computer  gssst  b§  assess* 
plish®&»  f&bssgasatly,  g@slti.aa  ilsiez^las&lsaa  sad  asesssasy  aiigassss© 
tiCKs  as®  «$airg&. 


8.  Sarth  entry.  ‘Th©  serth  entry  ee^aaao©  i@  bogus  %  jaiilsaniag  tbs  gg 
®ad  oriszstteg  tbs  SM  bass©  i&rms&  tef'er®  tbs  ©sgth*®  atsosghsra  la  rssehM,  Q^o® 
entry  late  _'  otsospkaro,  tbs  trls  attitude  of  the  mhiele  provides  a  ©eastasfc 
lift-teg  mti®  of  0.5*  Boll  cestrol  is  used  as  tfe®  sol®  ee&ns  of  ccntreXling 
th©  tmj©0tsgy~*toH«oti®s  in  tbs  direct!?®  of  tha  lift  vector  tesisg  obtained 
in  this  mgr,  kt  2*>?000  foot  the  ksat  shield  is  jotiiga&gd  md  a  tegea  afessts 
deployed,  sad  at  X5»OC30  feat  tbs  min  shade  is  deployed. 

It  is  evident  that  fhm$  critical  sissies  activities  involved  a  variety 
of  tssko.  These  say  ba  conveniently  categorised  as  flight  oosstrol,  switching 
mi  informtim  handling  (i*©,,  precatoss),  or  nsvigaticmal  ts$ks0  Flight 
control  and  switching  tasks  are  involved  in  all  of  the  aajor  activities,  hut 
the  navigational  task  ia  required  only  is  the  process  of  ssakksg  position 
dstersteatieBis,  la  this  8i!salaiien9  the  flight  control  ms  almost  astriotly 
Mftusa*  in  that  mtamtio  control  ms  liMted  to  aa  sttltadsHhold  function 
relative  to  fiaasd  inertial  spam  or  to  local  vertical  is  orbit,  CSxasg©®  in  th© 
attitude  of  either  vehicle,  initiation  sad  cutoff  ef  the  anginas,  rM  transit- 
iory  control  ?m*s  all  unto  direct  pilot  oesstrol.  Of  course,  memmsg  flight 
Inf  cassation  ms  sw&AXshle  through  appropriate  displays  and  throat  interrogation 
of  tk®  eJm&atsd  flight  computer,  O&sakllstB,  cusrtoaissd  by  the  crews  for  their 
o®  ®s®,  mm  available  to  and  used  by  ihm* 

J^teLiSl MM§I>  In  addition  to  the  mjor  activities  just  described, 
tk®  arm®  wsm  realistically  required  to  carry  out  other  act  lens  appropriate  to 
the  aimlated  sisalcm  ©ad  study  purposes.  Adjuaetual,  of  cousssj  to  th©  sain 
activities  they  had  certain  preparatory  cheeks  to  nsi.  Mac,  periodically  aal- 
ftoctioM  mm  introfiiiosd  for  shiah  esjas^enoy  prooctoeo  ms s  required,  Xso- 
mtrio  azorciscH  Khich  had  ba«ti  pmorticad  earlier  mm  aarfomM  periodically ^ 
Only  frroB9”driod  food  sas  proridsd  and  each  itea  each  ssoal  ms  rated,  Bio- 
ssdical  mamros  of  blood  prsssar®,  oral  tsg^?«mtimj,  aad  pula®  mta  mm 
regularly  taksa  by  the  individual  oroias^ihars,  Faoas  sad  urine  saapies  mra 
packaged  for  later  analysis*  While  cm  th@  lunar  surface  tbs  IS!  orm  identified 
rook  «as®l«c»  photographod  a  eisulated  esarfacso  display  and  described  it  orally 
and  in  writing.  tkrmmrt  such  activities  aa  these  era  not  of  direct  s®l®maoe 
to  th®  test  of  skill  ret  eat,  ion  presently  being  described,  beyond  serving  to 
fterthar  oharactcrica  th®  stellated  oiesioa  for  shieh  the  test  psrsonnel  had 
boon  trained, 

£g§yj^iLi^ialzia»  Each  of  tbs  tost  crews  received  5  «©ehs  of  cpscialisad 
t mining  for  tho  aisaalated  aisaica  and  than,  in  the  6th  vmk,  pcrfomwl  th» 
aimlated  aissica  in  real  tins,  Thin,  for  ths  purpose  of  the  retention  test, 
they  nay  ha  coanidared  to  hav®  rooeived  air  vseka  of  tradni ng,  the  last  s®sk 
being  tootod  to  a  oocrpleta  porforESsnoo  of  ths  actual  (sibilated)  aiesion. 

During  the  5  msks  of  pscaieBion  training,  a  ssaieuetosaised  training 
plan  mo  followed  to  achieve  aa  orderly  progression  of  skill  aoqulaitica.  In 
gsaeml,  ths  erevs  worked  an  8-hour  dsy,  5  d^ys  a  week  in  the  sisailator,  plus 
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about  2  houses  per  day,  6  days  a  wsok  at  physical  conditioning.  The  first  3  to 
5  dsya  isara  devoted  to  lecfcuras  (concerning  ths  mission,  the  vehicle  cystoma, 
end  the  displays  end  controls),  to  stu^r  of  mission  and  system  written  materials, 
and  to  oa?sw  dsvalcpasnt  or  rofinsstsni  of  their  checklists.  A  medical  exaaina- 
tion  was  given  sad  physical  conditioning  ms  started  at  that  time. 

Following  ths  initial  period,  each  omsaeaber  was  introduced  to  and  allowed 
to  praetioa  individually  each  main  Bission  task,  in  turn.  The  instructor  first 
demonstrated  ths  task  and  then  observed  and  critiqued  as  necessary  the  crow- 
E&Kbasr1  a  initial  perf© rsaaess  of  ite  Ecalietic  mission  communication  procedures 
wars  used  during  the  period* 

In  the  next  portion  of  the  training  sequence,  crewmembers  practiced  as 
whole  sequences  of  tasks,  tha  main  activities  involved  in  the  mission  (e.g„,  the 
lunar  landing  sequences).  Special  practice  on  particular  tasks  was  interspersed 
os  either  convenient  or  needed. 

Finally,  during  tha  last  5  to  7  days,  as  this  training  continued,  prepara¬ 
tions  for  th®  mission  in  the  fora  of  briefings  on  oheoklists,  food,  and  geology 
wsro  conducted  and  physical  status  assessed.  In  the  last  two  days  of  their  pre- 
mission  period,  0-4  and  0-9  performed  the  mission  in  fast  time,  with  the  coast 
phases  eliminated,  as  a  whole  orew.  This  permitted  some  adaptation  to  the  living 
arrangeasnts  end  miscellaneous  mission  requirements  aa  well  as  further  direct 
practice  of  mission  tasks.  0-8  and  0-13  performed  this  fast-time  mission  several 
days  earlier  in  their  training  sequence, 

Tha  simulated  mission  was  than  performed  in  real  time  over  a  169-hour 
period,  as  already  described. 

In  general,  both  pairs  of  crews  received  the  same  type  of  training.  How¬ 
ever,  the  crews  trained  later  (0-8  and  0-13 )  did  receive  substantially  more  actual 
practice  in  the  various  mission  phases  than  did  the  earlier  orews.  This  was 
largely  the  natural  consequence  of  continuing  experience  in  using  the  simulator 
and  improvement  in  operating  routine. 


In  the  test  of  skill  retention  the  orews  were  not,  however,  required  to 
perform  the  full  7~dsy  simulated  mission.  Instead,  in  the  interest  of  minimis¬ 
ing  coats  while  still  providing  an  adequate  test,  they  ware  required  to  perform 
the  mission  in  fast  tiaa— that  is,  with  the  long  translunar  and  transearth  coast 
periods  and  tha  position  determinations  and  midcourso  corrections  normally  per¬ 
formed  in  them  eliminated.  In  addition,  the  systems  and  log  oheoks  normally 
performed,  as  wall  as  tha  transearth.  insertion  phase  itself,  also  were  not 
required.  Sine©  the  interest  was  in  primary  mission  activities,  ths  other  activ¬ 
ities  originally  included  also  were  omitted.  These  omissions  made  it  possible 
to  ©capleto  the  fast  time  simulated  test  mission  within  about  13  hours  of  a 
single  workday,  while  still  allowing  as  before,  each  crewmember  to  perform  each 
major  activity,  Ii  this  way,  considerable  test  performance  data  were  gathered 
with  minimal  simulator  operating  time  and  cost.  The  planned  test  mission  sched¬ 
ules  ar©  reproduced  in  abbreviated  fora  in  tables  II  and  XU  to  illustrate  the 
testing  sequence.  Prom  these  it  may  be  noted  that  except  for  transposition 
by  0-4  and  0-9  pilots  within  crews  performed  the  test  in  an  invariant  order. 
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T&m  n 

Sehsdaia  for  0»4  a&d  0-9  Test  Missies 

ifiam^i^aMrt-tiAsn 


Phase 

Tim  (mm) 

Pilot 

Hot* 

lagj 

Briefing 

08®) 

-  Pilots*  Office 

5  — 

Pilot  Insertion  Mid  Preletmoh  Check 

0830 

1 

2 

3 

Barth  Aaooat 

0900 

1 

2 

3 

TraneXtm&r  Insort ion 

0930 

1 

2 

3 

Trsaslunar  Insertion 

0950 

2 

3 

1 

Traneluaar  Jncortioa 

1010 

3 

1 

2 

Transposition 

1030 

3 

1 

2 

Transposition 

1050 

1 

2 

3 

Transposition 

1120 

2 

3 

1 

UM  Status  Check 

1145 

1  (3*) 

lamer  Orbit  Insertion 

1230 

1 

2 

3 

Lunar  Orbit  Insertion 

1250 

2 

3 

1 

Lunar  Orbit  Insertion 

1310 

3 

1 

2 

Lunar  Leading  and  Ascent  - 
Hooking  with  <H 

1330 

3  (l*) 

(2*) 

Lunar  landing  and  Ascent  - 
Looking  with  CSS 

1530 

1  (2») 

f 

-  / 

Lunar  Las  dins  «ad  Accent  - 
Docking  with  (M 

1730 

2  (3«) 

(l*) 

I&rth  Satry 

1930 

1 

2 

3 

Earth  Entry 

2000 

2 

3 

1 

Earth  Swtry 

ago 

3 

1 

2 

Oc®pl©ti<®  /  Debriefing 

2100 

-Pilots*  Office 

- 

*  Positioned  is  laicar  firaareion  Moguls 
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BASS  in 


Schedule  for  <5»8  and  0“13  Tost  Mission 


Etaaft 

StaaJmzl 

?ilQt 

1 

iSi <£* 

Briofiag 

0730 

-  Pilots’  Office 

- 

Earth  isosnt 

0800 

1 

2 

3 

Traaslun&r  Jnear&ios 

0915 

1 

2 

3 

Transposition 

0935 

1 

2 

3 

LM  Status  Ohsck 

0955 

1  * 

3 

Lunar  Orbit  Insertion 

1035 

1 

2 

3 

Lunar  Leading 

1055 

1  * 

3  * 

Lunar  Assent,  Eandesvcus,  Looking 

1155 

1  * 

3  * 

Earth  Entry 

1255 

1 

2 

3 

Trsnslunar  Insarti.on 

1340 

2 

3 

1 

Transposition 

1400 

2 

3 

1 

Lunar  Orbit  Insertion 

1420 

2 

3 

1 

Lunar  Landing 

1440 

2  * 

1 

Lunar  Ascent,  Eandfasvous,  Booking 

1540 

2  * 

1 

Earth  Bitiy 

1640 

2 

3 

1 

Translunor  Insertion 

1725 

3 

1 

2 

Transposition 

1745 

3 

1 

2 

Lunar  Orbit  Insertion 

1805 

3 

1 

2 

Lunar  Landing 

1825 

3  * 

2 

lunar  Asosnt,  Honda  bvous.  Booking 

1925 

3  * 

2 

JL 

JL 

-1- 

Cosplotion  /  Debriefing 

2115 

-  Pilots’  Office 

«• 

r  Positioned  in  Lunar  Scours  ion  Modulo 


But}  those  of  0-4  sad  0-9  all  performed  s  given  mission  eegsont  before  going 
on  to  tha  nert  segsant,  whereas  those  of  0-6  and  0-13  par fonaad  the  ©stir© 
jaisoioa  In  eequanoe  boforo  changing  position.  Actually,  bscsuse  of  their 
effective  and  expeditious  perfonsaneo,  tbs  crews  completed  their  missions 
eossswhat  earlier  than  is  shown, 

Ga  the  (or  days)  following  that  in  whioh  ths  fast-tins  tost  mission 
vj&s  psrformdj  each  ores?  v&n  then  further  required  to  per form,  rspaatedly, 
selected  priory  mission  activities  of  special  interest.  As  in  the  test  mission, 
each  orewssaber  porfotmd  on  each  of  the  phases  so  that  data  were  obtains!  flswa 
all  the  participants.  The  purpose  of  these  trials  ms  to  obtain  a  basis  for 
estimating  hoe  rapidly  any  skill  loss  occurring  over  the  retention  period  sight 
b»  cosponsatsd  for  by  renewed  prsotice.  Also,  the  additional  training  afforded 
in  this  my  permitted  a  check  on  whether  the  ozma  had  attained  saxissua  efficiency 
at  the  end  of  original  training  (particularly  those  having  3  days  of  additional 
trials. 

For  two  of  the  crews,  receiving  1  day  of  repeat©!  trials,  tha  prisary 
mission  phases  sslootsd  were  transposition,  LEI  braking  and  hover,  I M  docking, 
treasaarth  insertion,  and  earth  entry.  For  the  other  two  ormv,  receiving  3 
days  of  reposted  trials,  the  lunar  orbit  insertion  m  substituted  for  the  traa b~ 
earth  insertion  and  the  LM  separation  and  doorbit  phase  ms  additionally  required, 
Th©  nmb®r  of  trials  given  accordingly  are  shown  in  table  IF,  for  the  two  pairs 
of  ororo,  respectively,  Baeesase  of  thsir  speedy  psrformaoe,  the  crows  war©  able 
to  perform  more  trials  item  had  boon  planned  originally. 


TABLE  17 


Eepaated  Trials  on  Mission  Phases 


Transposition  (THN)  2 

Lunar  Orbit  Insertion  (LOl) 

Separation  and  Beorbit  (SBO) 


Braking  and  Hover  (BE)  8 

Socking  (Bok)  6 

Traasoarth  Insertion  (TU)  2 

Earth  Entry  (S3)  5 


*  Ikeopt  for  P-81  who  perform!  only  8 


12 

12 

12  (16  for  P-83) 

16  (28  for  P-131 1 

24  for  P-132,  P-133) 

8  (12  for  P-132,  P-82,  P~83) 

12 

>,  12  BS,  and  8  E3  phases 
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The  facilities  and  equipment  used  in  this  test  were  those  used  in  the 
previously  mentioned  contract  research  on  human  reliability,  in  which  th®  train¬ 
ing  and  real  time  mission  were  performed,  The  major  components  employed  were 
a  simulated  send  nodule  with  associated  outside  displays*  a  simulated  lunar 
excursion  so  dale  or@w  oogpaxtKsoni  with  associated  outside  displays  %  extensive 
analog  computing  and  recording  equipment,  &  simulation  control  room  with  panels 
and  consoles  for  monitoring  displays,  making  inputs  to  them  and  recording  system 
outputs |  and  supporting  office  and  maintenance  areas.  A  main  simulation  area 
(about  1500  sq.  ft.)  contained  the  two  vehicle  simulations,  the  control  room 
ms  adjoining,  and  most  of  the  analog  computing  equipment  was  in  a  separate 
building  over  1000  feet  away. 

Command  Module.  The  simulated  CM  was  a  truncated  60°  cone  of  aluminum 
skin  on  stringer  and  frame  construction  having  a  bass  of  166  inches  diameter 
and  an  ©nolossd  volume  of  about  400  cubic  feet.  It  ms  oriented  with  the  small 
end  forward  and  the  axis  of  symmetry  horizontal  on  a  vibration-isolated  and 
sound-damped  cradle.  A  hatch  in  the  base  (or  rear)  was  normally  used  for  entry 
and  a  hatch  in  the  small  end  allowed  craw  transfer  to  the  LM?  vhr,n  attached  to 
the  CM.  Figure  1  depicts  the  vehicle  as  seen  from  the  right  rear  and  shows  the 
normal  entry  hatch  with  navigational  equipment  nearby.  The  front  cf  the  vehicle 
may  be  seen  in  the  background  of  figure  2. 


Figuro  1.  Eight-Sear  View  of  Command  Module 
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The  interior  apace  was  arranged  by  ssgane  of  suitable  part-floors  sad 
fixtures  into  several  areas.  The  duty  area  was  located  in  the  upper  formed 
portion.  It  consisted  of  a  sain  control  panel  stretching  completely  across 
the  vehicle,  small  aid®  panels,  and  three  seats  counted  sidc-^yside  facing 
the  panel*  The  isiddlo  seat  could  he  coved  forward  or  backward  or  removed 
completely  for  us©  in  tha  off-duty  area.  The  seats  were  allocated  to  the 
commander,  navigator,  and  engineer  in  left  to  right  placement.  In  the  upper 
rear  portion  ms  a  sanitation  area  for  waste  disposal  at  the  left  and  a  food 
said  water  area,  at  the  right*  In  the  lower  portion  were  a  sleeping  area  to  the 
left  (containing  &  mattress,  privacy  curtains,  and  separate  lights)  and  an  off- 
duty  area  to  the  right  of  a  central  hay.  A  navigator's  station  me  just  inside 
the  entrance  hatch  in  the  rear  bulkhead.  Three  TV  cassias  provided  separate 
monitor  views  of  the  three  crewmember  stations.  There  were  live  microphones 
in  both  the  duty  and  off-duty  areas  as  wall  as  individual  headsets.  Main  rocket 
engine  noise  was  simulated  through  a  separate  speaker  above  the  duty  area.  The 
interior  of  the  vehicle  was  lighted  and  air  conditioned  by  integral  units. 

At  the  pilot's  station  (figure  3)»  in  addition  to  appropriate  function 
switches  and  status  indicators,  were  the  main  instruments  and  controls  for 
controlling  the  vehicle.  A  eldest ick  for  translation  was  located  on  the  left 
armrest  and  another  for  attitude  control  was  located  on  the  right  armrest. 
Deflection  of  the  translation  control  resulted  in  proportional  translatory 
accelerations.  Motions  in  X-  and  T-dimensions  produced  compatible  forward  or 
sideward  accelerations,  respectively,  and  clockwise  twisting  resulted  in  down¬ 
ward  accelerations.  Inflection  of  the  attitude  control  produced  proportional 
angular  velocities  in  the  same  directions  as  a  conventional  aircraft  stick  and 
th8  additional  tv.ah  motion  produoed  yaw  in  the  direction  of  twist.  A  3-axiB 
ball  indicator  with  pitch  and  yaw  error  needles  provided  primary  attitude  informa¬ 
tion.  A  digital  indicator  was  provided  for  setting  in  desired  velocity  incre¬ 
ment  a,  as  in  orbital  insertions  and  corrections.  This  would  pulse  down  to  zero 
as  the  thrust  was  introduced,  at  which  time  the  pilot  was  to  cut  off  the  engine. 
For  transposition  a  3-needlo  display  indicated  the  information  on  CM  position 
relative  to  the  LEI  necessary  for  the  pilot  co  translate  back  to  it,  by  oo®” 
pensating  for  the  indicated  errors.  For  reentry,  a  2-needle  roll  meter  indicated 
the  roll  program  and  actual  roll  superposition  of  the  two  needles  being  the 
continuously  desired  status.  Actually,  since  this  status  was  virtually  impos¬ 
sible  to  achieve,  deliberate  uncomnanded  roll  corrections  were  required. 

At  the  navigator's  station  (figure  4)  war®  mounted  a  single  line  of 
eight  scanning  telescope,  a  dual  line  of  sight  sextant,  a  small  2-axis  stick 
(for  proportional  control  over  shaft  and  trunnion  angular  rates  of  the  tele¬ 
scope  and  one  sextant  line  of  sight  relative  to  tho  other  sextant  line  of 
sight),  a  small  3_ox±8  stick  for  "bang-bong"  control  of  attitude  drift,  and 
associated  switches.  Outside  the  module,  operated  in  conjunction  with  the 
navigator's  equipment  was  a  g label-mounted  ascea^lv  of  a  starapher®  and  slide 
projectors.  With  those,  a  atarfield  (110°  visible)  and  one  of  up  to  six 
different  earth  or  coon  views  were  presented  on  a  7-l/2-feot  spherical  screen 
as  a  display  for  navigational  fixes  and  guidance  system  alignments  in  oislunar 
space.  The  navigator's  attitude  drift  control  moved  the  entire  soon©  in  yaw, 
pitch  or  roll  at  very  slow  rates  through  the  giabal  drive  poohanisna. 
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Figure  4,  Eavigator»s  Station 


The  entire  SC  simulation  was  programmed  on  one  Pace  231H  Computer  using 
several  economization  techniques.  Because  of  the  commonalities  in  equations 
for  orbital  insertions*  aidcourss  corrections,  transposition  and  earth  entry, 
it  was  possible  to  program  the  rotational  equations  and  change  of  velocity 
calculation  onoe  for  all  these  phaaeB.  With  this  arrangement  and  additional 
programed  calculations,  aa  required  (for  earth  entry,  in  particular),  the 
change  from  one  phase  to  another  was  accomplished  merely  by  changing  certain 
potentiometer  settings  (e.g. ,  mass,  mass  flow  rate,  thrust,  moment  of  inertia, 
control  gain).  Wo  more  than  12  changes  ware  required  in  any  circumstance. 

Earth  ascent  and  position  determination  required  unique  programs,  but  relatively 
little  computational  equipment  by  comparison.  Then,  additionally,  to  obtain 
desired  informal  Lon  on  pilot  performance  from  flight  parameters  and  manipula¬ 
tions  of  them,  the  program  was  separated  into  two  patch  panels.  One  contained 
all  phases  to  earth  entry  and  th®  other  contained  only  th®  earth  entry  phase. 

These  were  arranged  in  such  a  way  that  different  potentiometers  war©  used,  thus 
enabling  a  switch  from  one  to  th©  other  without  changes  in  potentiometer  settings. 
Appropriate  checkout  and  calibration  techniques  war®  used  following  changeover 
fra  one  phase  to  another  to  insure  a  oorroot  simulation.  The  computer  controls 
were  wired  so  that  appropriate  portions  of  the  program  automatically  became  active 
whenever  the  pilot  selected  a  given  control  condition,  and  terminal  values  at  th® 
completion  of  the  task  wer®  automatically  held  for  readout. 


M-l  J&gg  _  .  Tha  simulated  orow  cosparteant  of  alurainna 

skin  on  stringers  and  fimea  provided  sufficient  space  for  sido-by-aida  seating 
of  two  arcs  ambers  facing  forward  in  saaioraot  posture.  The  seats  consisted 
of  haxassssB  attached  to  vertical  cradle b  of  tubing*  A  forward  hatch,  permitted 
aooass  frees  and  return  to  the  <U  when  tbs  LSI  was  attach©*.  The  oompsrtsssjit  mo 
supported  at  th®  cantor  of  gravity  by  a  3~®sl8«  hydraulically  driven  giabal 
systea  that  permitted  attitude  changes  of  4  40°  in  pitch  and  roll  and  4  180°  in 
yaw  in  response  to  coBwrtsor  signals.  (Actually,  pitch  and  roll  ware  used  only 
in  looking  to  the  Halts  of  4  15°»)  The  gisb&l  sysrtm  ms  supported  by  a  stool 
stand  of  such  height  that  ih®  hinge-point  ms  8-3/2  foot  above  floor  level,  Tha 
front  or  hatch  end  of  tha  LSM  or®w  ooEparfcaczrt  say  bo  soon  in  th©  aiddlsgrcund 
of  figure  2,  uhioh  shows  ths  oonpartssnt  just  ahead  of  the  simulated  CM,  tfeich 
is  at  the  roar. 

At  ths  pilot's  station  the  primary  controls  provided  for  flight  control 
were  aa  attituds  stick,  a  translation  stick,  and  a  sain  engine  throttle.  Th© 
attitude  atiok  was  fLoor^sauatod.  Inflection  of  it  in  a  given  direction  produced 
proportional  angular  rates  in  pitch,  roll,  and  yaw.  Esarsard  deflection  restated 
in  pitoh-up,  movement  to  the  right  produced  yaw  in  th©  earn©  direction.,  and 
ooantorolookwise  twisting  produced  right  roll.  To  operate  th©  stick  it  w 
naoossary  to  depress  a  trigger  which,  when  released,  returned  th©  system  to  an 
automatic  attitude  hold  condition,  Tha  translation  control  teas  a  T-srtick  project¬ 
ing  horizontally  froa  ths  inatrusant  panel.  In  on©  aode  used  for  dsorbit, 
rondosvoua,  and  docking,  deflections  produced  proportional  translator?  accelera¬ 
tions.,  In  the  other  sod®,  used  for  honoring,  deflection  produced  proportional 
translator?  rates.  O^wardrdotsnsard,  right-left,  and  formrdr^re&rward  motions 
which  it  permitted  all  resulted  in  translation©  in  ths  corresponding  directions. 

A  trigger  on  the  translation  stick  coupled  th©  throttle  to  it  and  looked  the 
simulated  vehicle  in  a  hover  while  still  enabling  translations. 

Ths  primary  displays  included  a  AV  counter,  a  combined  erossrssg®  and 
oroasrange-rate  aster,  a  combined  attitude  and  attitude  rat©  motor,  a  range 
meter,  a  range  rate  meter,  and  a  CBS  display  of  downraago  and  cxosanango  die- 
plaoeaant  fcca  th©  landing  site.  As  in  the  a  isolated  (Si,  to  add  velocity  the 
pilot  set  the  computed  value  on  a  digital  meter,  fired  the  ©agin©  at  the  proper 
tia©  and  cut  off  the  engine  when  th®  indicated  valuo  reached  zero.  Th®  range 
and  range  rat©  meters  were  interpreted  with  th©  aid  of  a  table  attached  to  the 
panel  which  showed  tha  schedules  of  rang©  rate  with  range  for  various  throttle 
settings  producing  a  vehicle  stop  at  the  landing  site.  The  attitude  control 
display  wae  a  large  meter  indicating  attitude  by  position  of  a  long  horizontal 
needle  and  attitude  rat©  by  position  of  a  short  vartiol©  naodl® — the  optimum 
continued  intersect  of  th©  two  for  various  braking  attitudes  being  shown  by 
appropriate  curves.  Similarly,  on  ascent  profile  on  th®  ossa  motor  guided  th© 
powered  ascent.  In  addition  to  the  GST  display  of  downrange  and  orossrange 
displacement,  a  downrange  rat©  needle  was  also  included  on  ths  orossrauge  meter 
to  facilitate  ths  hovering  and  landing  teaks. 

A  spherical  screen,  two  projector  assemblies,  and  a  translator  were  used 
to  provide  realistic  outside  views  in  th©  various  phases  of  the  lunar  excursion. 
Separate  triangular  windows  in  th©  crow  oocpartcaat  allowed  each  orcssaeabar  a 
viaufiald  4  90®  in  azimuth  and  18  to  -90°  in  elevation.  Th©  appropriate  views 
wore  simulated  by  project  ions  on  a  24-foot  diameter  screen.  A  star-horizon 
projector  mounted  on  a  3-axis  global  system  on  the  ceiling  of  the  simulator  room 
and  driven  by  ooaputor  signals  ^ar©  realistic  indications  of  pitch,  roll,  yaw, 
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central  angle  and  altitude.,  Th©  oiarspherts  o f  18-inoh  diameter  included  tha 
2300  brightest  stare,  A  small  incandescent  Issrp  and  suitable  saaks  iwieM 
outside  it  gave  the  illusion  of  th©  soon  and  earth  end  a©oa  herisons.  For 
rondo evens,  a  separate  pro^ootor  ssusfesd  on  a  S-asis  gisb&l  support  end  oriented 
by  computer  signals  derived  from  thos®  raprassatiag  the  orbits  of  the  vehicles 
(iffl  mtd  CH)  gave  a  realistic  view  of  ths  (SS  outlined  by  its  fleshing  b$®e*m 
which  beosma  e  steady  out  1  ins  increasing  in  sis®  with  disiais&lni  range.  Coin¬ 
cidentally,  as  rang®  approached  12  foot,  through  nenapparent  3/32-inoh  perfora¬ 
tions  (spaced  l/2-iaoh)  in  the  b croon,  the  crew  viewed  an  illuminated  translator 
consisting  of  a  lighttight  shell  of  th®  sa@®  shape  as  ths  (H.  Then,  for  dock¬ 
ing,  tha  sorsen  was  rapidly  separated  along  a  vortical  interline  m&  the  two 
halves  rotated  backward  and  to  th©  elds  by  psasussatio  pistons  end  hydraulic  actus?* 
tors*  With  th®  1M  <3C8partssst  gisabals  peraitting,  pitch,  roll,  and  yaw  sot  ions 
and  the  translator  mounting  p©raittisg  vertical  and  lateral  travels  up  to  £  5 
foot  and  fore-aft  travel  up  to  12  feet,  physical  docking  ma  xbm  accomplished. 
Translations,  by  msans  of  direct  current  olectrio  ©©tor  servo  systems  sad  cable 
drives,  were  signaled  'by  the  oesputer  on  ths  basis  of  pilot  dsflectioa®  of  the 
LEM  translation  stick.  The  (SI  ms  assumed  stabilised  in  relative  position.  The 
simulated  I®  cospsrts&snt,  ths  screen  (partly  open),  and  th©  translator  (CM)  say 
be  seen  in  the  center  to  rsargreund  of  figure  5»  Just  ths  sidewall  of  th® 
siaulatad  ffii  shows  in  the  foreground  of  this  view. 

The  I®  siiaulation  required  the  use  of  two  Paoa  23 IE  Cespotera  aad  con¬ 
siderable  associated  switching  and  relay  circuitry.  Bscaoee  of  dissimilarities 
in  parameter  ranges,  ccssoand  inputs,  etc.,  the  mission  was  divided  into  a  ©cries 
of  individual  problems  or  phases.  Also,  because  of  the  nature  of  th®  operation, 

*  two  axes  of  references  are  required*  an  orbital,  with  translating:  origin  and 
nonrotating  axis  for  dsorbit  and  rands svous}  and  suborbital,  with  nontranslaticg 
origin  for  braking,  landing  and  ascent.  Th©  equations  of  notion  mro  progressed 
only  once  and  stepper  switches  were  used  to  go  from  one  phase  to  ths  next.  Ikoepi 
for  the  inclusion  of  attitude  hold  circuitry  and  rotation  of  the  roll  axis  to 
90°,  th©  rotational  equations  vera  performed  in  the  conventional  way.  In  pro¬ 
gramming  tha  translation  equations,  because  of  the  display  range  requirements, 
careftil  attention  was  , given  to  the  problem  of  scale  changes.  These  were  accom¬ 
plished  by  ©witching  tha  displacements  at  th©  rescaling  point  of  the  rang®  dis¬ 
play  and  by  alternately  using  parallel  integrators  and  storing  the  variable 
values  at  a  scale  factor  10  tines  higher  than  th©  on©  being  used.  Th®  long 
coasting  descent  and  ascent  phases  war®  function-genarated  rather  than  computed. 

As  with  tha  CLi  simulation,  tha  computer  controls  vsra  wired  so  that  terminal 
values  at  th©  completion  of  th©  phase  a  wore  automatically  held  for  readout. 

Th®  entire  simulation  ms  monitored 

and  coordinated  from  an  adjacent  control  room,  a  view  of  which  is  shown  in 
figure  6.  Included  war©  a  communicator9 s  consol©,  a  flight  director* »  consol®, 
systcaa  operation  consoles,  and  data  recording  equipment.  An  interoon  eystco 
connected  th®  control  room  with  the  a  isolated  vehicles  tod  computer  personnel. 

It  provided  for  Benito  ring  of  all  oossamication  by  control  and  computer  porson- 
nel,  tape  recording  of  all  vehiele-oontrol  room  cosmmioat ions  daring  missions, 
and  separate  oesmuni cations  with  the  observer-operator  in  the  min  eicnlation 
room.  Vehicle-control  ooeramioations  could  be  delayed  as  a  function  of  range. 

A  direct  telephono  line  waa  available  batcocai  tha  flight  director  and  oesputer 
personnel.  Tha  eossamieatox  bad  available  th©  mces&axy  ooEsanlostioa  equipment, 
a  mission  tiee  indicates* ,  duplicate*  of  all  (M  and  IM  caution  and  warning  indi¬ 
cators,  and  a  TV  monitor  (suitohabl©  to  any  of  the  throe  ®  crow  position 
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monitors  or  a  camera  viai dug  the  LM  docking.  The  flight  director  had  amil- 
able  a  similar  T7  monitor;  duplicates  of  moat  <M  and  LSM  flight  inatrusants; 
duplicates  of  all  CM  and  13U  mrning  and  caution  indicators;  indicators  ml 
switches  relating  to  control #  navigation  and  mdar  systems;  and  controls  for 
the  mission  timer  end  outside  projection  Airplays.  Susercus  systems  operations 
consoles  provided  indicator  lights,  switches;  and  potentiosseiars  oorroapsadiEg 
to  lights,  switches,  and  meters  on  the  simulated  vehicle  panels.  The  so  allowed 
full  aonitorahip  of  the  flight  crew’s  operation  end  appropriate  responses  to 
it  as  well  as  the  introduction  of  malfunction  indications. 

Within  the  control  roc®,  three  100-ehaaansl  Brush  binary  recorders  wore 
connected  to  either  the  simulated  CM  or  LEM  compartment  to  record  running 
information  on  switch  positions  and  status  indicator  lights.  A  fourth  was 
connected  to  the  CM  inflight  test  system.  Honflight  system  meter  readings, 
motions  of  the  navigation  optics,  and  forces  exerted  during  the  iaonotrio 
exercises  ware  recorded  with  three  50-shannel  Consolidated  Electro  dynasties 
oscillographs.  In  the  computer  laboratory,  flight  measures  were  recorded  on 
five  Mark  200  8-ohaonel  rectilinear  recorders  and  two  11  x  l?~inch  X-T  plotters 
and  digital  printouts  mads  of  relevant  terminal  conditions  on  the  computers. 

In  view  of  the  ©aphasia  upon  assessing 
task  skill,  as  such,  tue  simulation  was  considered  to  be  reasonably  adequate 
for  the  purpoeo  of  this  study.  This  may  be  evident  from  the  brief  description 
just  given  of  the  instruaantaticn,  although  the  reader  may  wish  to  check  on 
certain  details  in  the  report  by  Grodaky,  Mandour,  et  al  (1966b}  or  others  to 
which  that  report  makes  reference.  However,  certain  rather  obvious  deviations 
from  a  full,  high-fidelity  simulation  should  he  noted  for  the  record. 

First,  cues  and  conditions  deriving  from  motion  war®  simulated  in  a 
limited  way.  Although  attitude  and  transl&tory  motions  could  ho  Kid©  up  to 
reasonable  limits  for  the  purpose  of  specific  ssuieirvere,  there  sis  no  simula¬ 
tion  of  ascent  and  reentry  accelerations  and  vibrations  or  of  impact  0  on  land¬ 
ing.  The  condition  of  weightlessness  also  was  not  simulated.  Second,  in  the 
life  support  area  the  environment  deviated  significantly  from  the  real  one  in 
that  appropriate  variations  in  atmospheric  ooaposition  end  pressure,  toxic  gases, 
and  radiation  were  not  included.  Also,  a  fully  realistic  handling  of  nutrition 
and  sanitation  was  not  required  nor  was  the  use  of  personal  protective  equip¬ 
ment  required.  Third,  with  re  upset  to  specific  tasks  the  lunar  landing  was 
made  with  reference  solely  to  instruments  rather  than  with  at  least  part  use 
of  an  cut-th«r-window  view.  Simulation  of  the  cut-the-windov  view  was  conoids  rod 
unnecessarily  expensive  in  view  of  the  study  purposes.  Finally,  the  contest 
of  mission  actuality  was  necessarily  missing  with  the  likely  result  that  crew¬ 
member  motivations  and  emotional  responses  (particularly  anxiety)  were  some¬ 
what  different  than  may  be  expected  on  a  real  mission. 

Just  what  *ffeot  these  departures  from  realism  my  have  had  on  the  out¬ 
come  of  the  study  is  difficult  to  guess.  There  are  reasonable  bases  for  arguing 
that  apeaifio  .ask  performance  may  have  bwen  selectively  improved,  or  select ivoly 
degraded,  or  err? as  unchanged  by  these  circumstances}  and  specific  cases  may  be 
made  and  supported  by  previous  studies  of  these  variables.  However,  it  is 
doubtful  shat  ha  1  overall  performance  on  a  real  mission  of  this  nature  would 
exceed  that  displayed  in  tbs  test.  The  principal  advantage  accruing  in  the  real 
mission  is  considered  to  ho  the  resulting  motivation  to  perform  wall  and  without 
failure.  All  the  teat  personnel  gave  evidence  of  such  an  intent.  Certainly, 


22 


thair  response  to  the  s insulation  was  generally  favorable — dhe  principal  excep¬ 
tion  bai ng  the  sriiiciss  ty  C-4  and  0-9  of  the  unconventional  fly-froa  arrangs- 
©sat  of  th©  landing  site  displaes&sat  indicator  in  th®  aimlated  ISM.  crew  compart¬ 
ment  vhiek  caused  confusion  in  direction  of  control  sot  ions,  Consequently,  the 
directional  response  of  this  indicator  was  reversed,  so  as  to  be  conventional, 
for  0-8  and  0-13, 
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The  details  for  arrangement  and  conduct  of  retention  testing  war©  simple 
and  straightforward.  Saving  previously  been  contacted  informally  concerning 
their  availability  end  warned  of  the  probable  requirement  for  their  presence, 
the  test  personnel  were  subsequently  directed  by  message  from  Air  Force  Systems 
Command  to  be  present  for  the  specified  period— 3  days  for  0-4  and  0-9,  5  days 
for  0-8  and  0-13. 

Accordingly,  and  in  keeping  with  informal  word  passed  to  them,  each  craw 
arrived  at  the  test  site  for  duty  .just  after  noon  their  first  day.  As  scheduled 
in  the  first  phase,  the  first  crew  to  be  tested  (0-4)  arrived  cm  a  Monday  and 
concluded  test  duties  Wednesday  evenings  the  second  orew  (0-9)  arrived  on  the 
seme  Wednesday  and  concluded  test  duties  Friday  evening.  All  testing  in  this 
phase  was  conducted  during  the  5  days  of  a  single  workweek.  As  scheduled  in 
the  second  phase,  th©  first  craw  to  be  tested  (0*»13)  arrived  on  Monday  end 
completed  work  the  following  Friday  and  the  second  crew  (0-8)  arrived  the 
succeeding  Monday  and  completed  work  the  succeeding  Friday.  There  was  one 
exception  to  this  general  arrangement  in  that  P-8l  of  0-8  was  obliged,  because 
of  unforeseen  other  duties,  to  leave  at  noon  Wednesday  rather  than  to  complete 
the  workwsak. 

Upon  arrival  at  the  teat  site  for  duty,  each  orew  was  given  a  briefing 
by  Air  Force  and  Martin-Marietta  personnel  about  the  purpose,  general  arrange¬ 
ments  for,  and  manner  of  conducting  the  test.  Following  this,  each  orew  was 
allowed  to  choose  how  it  would  review  the  systems  and  tasks  in  the  telan.ee  of 
time  available  that  afternoon.  Their  checklists  were  available  to  them  at  that 
tiffi®  and  eubaequently.  After  hearing  a  short  review  briefing  on  systems  tasks 
and  descriptions  of  any  change  in  the  simulation  (these  were  trivial),  0-4 
spent  about  l/2  hour,  0-9  spent  about  2  hours  and  0-8  and  0-13  each  spent  about 
1  hour  reviewing  their  tasks  and  procedures.  However,  the  crews  were  not  per¬ 
mitted  to  view  the  actual  displays  and  controls  or  enter  the  simulators.  This 
completed  the  first  day's  test  activities  and  crown embers  wore  then  free  to 
pursue  their  own  oonoems  until  the  following  morning. 

On  the  second  day,  each  orew  then  performed  the  simulated  fast  time 
mission  discussed  earlier  and  outlined  in  tables  II  and  III.  Qn  completion 
each  was  once  again  free  of  duties  until  the  following  day.  On  the  third 
day,  and  tho  following  two  days  for  0-8  and  0-13,  each  orew  then  repeatedly 
performed  selected  mission  phases,  as  shown  in  table  IV.  Oi  completion  of 
these  at  the  end  of  the  workday,  they  ware  given  a  very  short  briefing  on 
general  outcome  of  the  tost,  encouraged  to  make  any  ooaaents  they  chose  con¬ 
cerning  the  tost  and  their  test  performance,  and  then  released  to  return 
home. 
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Throughout  tha  steuiated  mission  and  mission  pnaee  performances,  appro- 
pri&ta  control  of  the  a  isolation  ms  maintained  by  ~<mtrol  rocg  porsesm©!  sad 
realistic  ooassuaications  procedures  were  used.  Certain  other  MartinrS&rietta 
and  Air  Force  personnel  also  m@  preasst  periodically  to  zsoaitor  tha  c oadaot 
of  the  test,  but  they  did  not  interact  with  the  pilots  during  actual  tssi 
performance. 
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laving  ccaprassed  the  normal  7-day  sdsaion  iat  a  single  workday  by 
omitting  the  miscellaneous  activities  normally  required  and  the  long  coast 
phases,  including  the  navigational  tasks  of  those  pi  jtsss,  the  measurable  aspeots 
of  performance  ware  considerably  reduced.  Nevertheless,  there  were  a  number  of 
measures  still  obtainable,  such  as  the  various  system  parameters  reflecting  upon 
flight  control,  the  associated  switching  requirement nt  and  the  procedural  checks 
preceding  and  following  various  phases.  However,  in  the  retest  performance  of 
0-8  and  0-13  on  the  days  following  iho  test  nission  ©wen  the  procedural  and 
switching  activities  war©  eliminated  in  order  to  obtain  additional  data  on  the 
apparently  more  sensitive  flight  control  capability.  The  not  result  of  those 
constraining  decisions  was  to  liait  tha  primary  utility  of  the  data  to  tho  area 
of  flight  control.  Jbirtb^raore,  as  do  scribed  by  Qrodsiy  ©t  ai  (1966a)  and 
Grod&fey  st  al  (1966b),  the  data  on  switching  and  procedures  do  not  show  evident 
sensitivity  to  the  retention  variable.  Also,  being  of  a  frequency  nature  and 
subject  to  various  fractionations  as  to  criticality,  phase,  position  and  so 
forth,  it  Sosas  unlikely  that  valid  inferences  could  "be  "based  upon  the  few  oasos 
in  eaOh  properly  distinguished  subset.  On  the  conclusion,  then,  that  h'jso  data 
contribute  little  to  the  understanding  of  skill  retention  in  this  inet  .  they 
are  not  treated  in  this  report,  but  they  are  summarised  for  the  eurienn.  Jar 
in  appendix  I. 

With  respect  to  flight  control,  of  coarse,  &  largo  nnsbar  of  measurable 
parameters  were  recorded,  and  son©  limitation  as  to  which  would  be  considered 
was  neoescory  to  keep  the  analysis  and  interpretation  tasks  ssnags&bl®.  Born, 
having  enjoyed  the  opportunity  to  cosss3nt  oonoorniag  the  various  Eea^juresent 
possibilities  before  they  were  seleoted,  it  ©eased  appropriate  to  use  in  this 
analysis  of  retention  approximately  the  same  measures  agreed  to  by  Martin- 
Mai  ietta  and  NASA  for  use  in  the  closely  related  hns&n  reliability  program 
( Contract  NASw-ll87).  This  group  of  measures  had  been  carefully  selected  from 
the  many  possible  with  the  intent  of  providing  a  sufficiency  of  information 
for  adequate  representation  of  the  flight  control  performance  and  their  general 
significance  can  hardly  be  questioned.  The  use  of  essentially  tha  ease  measures 
also  allows  for  more  direct  crass-oosapari&on  of  findings  in  the  two  studies  and 
of  results  obtained  via  different  analytic  ssthods.  Accordingly,  flight  control 
and  operations  wore  represented  by  the  various  measures,  as  briefly  described 
in  the  following  paragraphs.  The  associated  hypothetical  system  criterion  for 
each,  arrived  at  in  the  some  way  and  in  view  of  early  data,  is  also  included, 

TraR9l\m^«_toar  Orbit  .-.aM JtegB83M&.  J^la^&JSSLL* 

In  these  throe  insertion  phases  tho  aspects  of  performance  considered  nost 
critical  are  the  velocity  out— off  and  the  adequacy  with  which  the  pitch  program 
ia  t rooked.  Any  error  in  initial  orientation  of  the  vehicle  {(*&)  would  be 
promptly  nulled  out  as  a  pit oh  error,  engine— ignition  may  occur  within  a  time 
span  of  30  seconds  without  difficulty,  and  coasting  attitude  is  not  oritioal. 
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Since  a  speoific  velocity  was  required  at  cutoff,  ae  indicat.d  'by  a  counter 
pulsing  to  zero,  It  was  appropriate  to  simply  record  ir*  feat  per  second  the 
deviation  from  the  required  value.  Similarly,  since  pitoh  control  adequacy 
was  reflected  in  minimal  deviation  from  an  optimal  program,  the  differential 
was  integrated  over  time  to  indicate  overall  accuracy.  Hcsrsver,  accuracy  can 
be  reflected  in  two  obvious  ways— as  average  or  arithmetic  mean  deviation  and 
as  a  variation  or  standard  deviation  about  that  mean.  Hence,  both  the  man 
pitch  error  (X)  and  the  standard  deviation  of  pitoh  error  (a)  ware  recorded. 

In  the  analysis  presented  in  the  following  these  have  been  combined  on  a  trial” 
by-trial  basis  to  provide  an  estimate  of  the  maximal  error  occurring  95$  of  the 
time  (^.95))  assuming  the  error  distribution  normal.  The  actual  relationship 
used  is’7 

X  ®  X  +  1.6449s 

In  this  way  pitoh  control  accuracy  was  represented  by  a  single  measure.  The 
criteria  selected  were  £  10  fps  for  velocity  and  -  0.1°  for  both  pitoh  error 
mean  and  standard  deviation,  or  -  0.2645°  for  the  combined  pitch  error. 

Transposition  (TEN).  Performance  of  the  transposition  phase  ms  considered 
beat  reflected  in  the  displacement  and  displacement  rate  of  the  two  vehicles  at 
docking,  the  closing  rate  at  impact  of  the  two  vehicles  and  the  fuel  required 
for  accomplishment  of  the  phase.  Initial  separation  and  stabilization  war® 
accomplished  automatically  and  any  error  in  inversion  to  180°  could  easily  he 
corrected.  Hence,  displacement  (in  feet),  displacement  and  impact  rates 
(in  fps),  and  fuel  (in  slugs)  were  recorded.  However,  in  the  following  analysis 
displacement  was  omitted  from  consideration  on  the  argument  that  if  the  phase 
’is  accomplished  at  all,  the  amount  of  displacement  would  be  acceptable.  The 
corresponding  criteria  selected  ware  1  fps  for  both  rats  measures  and  10  slugs 
of  fuel. 

Separation  and  Deorbit  (S1X>)«  Separation  of  the  LEM  from  the  CM  and 
subsequent  deorbit  in  the  LEM  constitute  the  first  main  phase  in  the  lunar 
excursion  sequence.  The  most  critioal  aspects  seen  to  be  the  proper  orienta¬ 
tion  at  the  time  of  ignition  said  timely  engine  cutoff  upon  achieving  the  r^per 
change  in  velocity.  Accordingly,  the  measures  selected  as  of  special  interest 
were  velocity  cutoff  error  (in  fps),  pitoh  angle  error  (in  degrees)  and  yaw 
angle  error  (in  degrees).  The  corresponding  criteria  were  set  at  2  fps  and 
*  2°  and  *  1°,  respectively. 

Brake  and  Hover  (BH).  After  fc^a  deorbit  and  subsequent  coast  descent 
(in  which  no  piloting  is  required)  the  braking  and  hover  phase  of  the  lunar 
landing  must  bo  accomplished.  This  is  naturally  a  oritioal  phase  and,  at  least 
in  this  simulation,  a  complex  one  having  a  number  of  measurable  aspects.  As  a 
result,  a  varioty  of  measures  wore  selected  for  special  attention.  These  in¬ 
cluded  the  percentage  of  available  fuel  consumed  (an  efficiency  measure),  the 
displacement  (in  feet)  from  intended  touchdown  site,  the  vertical  translation 
or  impact  rate  (in  fps),  the  lateral  translation  or  displacement  rate  (in  ffce), 
the  roll  nr.d  pitoh  angles  (in  degrees)  and  the  attitude  change  rates  (in  degrees 
per  second)  in  all  axes  at  touchdown.  However,  in  the  following  analysis  several 
of  these  measures  ware  omitted  as  less  important  for  dose  scrutiny  and  the 
performance  is  represented  simply  by  fuel  consumption,  displacement ,  and  die- 
placement  and  impact  rates.  Th®  corresponding  criteria  are  95$  ?  200  ft,  and 
10  fps  for  both  rate3. 
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Lunar  (M)*  The  posared  &sc«nt  phase  is  relatively  alsple.  Eagles 

ignition  and  espsal®  orientation  to  mom t  altitada  present  no  probisa.  Track¬ 
ing  tbs  altitude  profile  by  varying  pitch  angla  w»a  not  particularly  difficult 
baewms®  of  tbs  slow  rat®  of  change  involved  nor  «aa  euhsiaqusnt  raoriantation 
at  1,000  fps  velocity.  Howsrar,  proper  cutoff  of  the  ©sgiss  at  th©  dsairsd 
velocity  is  critical  and,  tmdsr  these  circumstances,  e.pparsnily  subject  to  error 
resulting  frees  attention  to  other  asp acts  of  flight  control,  Therefore,  velocity 
cutoff  error  (in  fps)  was  iaksa  to  represent  performance  adequacy  in  this  phase 
and  the  criterion  -gas  sat  at  10  fps.  Tbs  following  coast  ascent  required  only 
tha  sasy  task  of  mintaining  radar  look  on  the  GS. 

Rtmd&ggvou.s  (Rend).  Upon  reaching  an  approximation  of  ths  GE!  orbital 
altitude,  rendezvous  with  it  was  accomplished  by  acquiring  a  visual  fix  on  its 
beacon  and  stabilising  in  a  cocrbiting  position  relative  to  it  by  operating  th© 
LEM  translation  stick.  This  performance  vas  considered  bast  represented  by  the 
efficiency  ssaffore  of  percentage  of  available  fuel  coneusad  in  the  process,  and 
the  criterion  sag  again  set  at  95$* 

Hooking  (Xtok\  Lika  the  earlier  transposition  phase,  tbs  sub&equoat  dock¬ 
ing  or  physical  join-up  of  the  LSI  with  the  €Si  is  reflected  in  the  critical 
performance  aspects  of  displacosssst  rad  displacement  rate  and  in  i&paot  rate. 
These  wore  duly  considered  in  the  analysis  described  in  the  following,  the 
criteria  being  1  ft,  0.5  fps,  and  0.1  fps,  respectively,  (iiiaplaocsssnt  yaa 
retained  in  this  case,  partly  because  no  independent  se&sur®  of  efficiency,  as 
fbel  oonsusption,  was  taken. )  Additional  measures  selected  and  recorded,  but 
not  included  in  this  analysis,  were  yaw  at  pitch  angle  errors  (in  degrees)  and 
rata  errors  (in  degrees  per  second)  in  alx  axes. 

ferth  Futrv  fffS),  The  earth  entry,  as  perforated  in  this  simulation,  again 
imposed  fairly  complex  requirssaanta  which  were,  in  turn,  reflected  in  a  variety 
of  measures.  Initially,  proper  orientation  had  to  be  assuaod  and  this  was 
represented  by  attitude  angle  error  (in  degrees).  Then  for  the  first  210  seconds 
tracking  of  a  roll  program  of  rssrps  and  flats  was  critical.  Psrforsimee  of  this 
was  again  represented,  like  pitch  tracking  in  the  insertion  phases,  by  the  aver¬ 
age  and  the  standard  deviation  of  error  during  the  period.  Next  altitude  and 
altitude  rate  errors  and  later  croo Grange  and  orosarange  rate  errors  ware  con¬ 
trolled  bf  appropriate  deviations  fron  the  cotaaandsd  roll  program.  Performance 
in  these  respects  was  considered  best  represented  by  average  and  standard  devia¬ 
tion  of  error.  Overall  accuracy  of  the  entry  was  represented  by  the  terminal 
displacement  from  the  landing  site  and  efficiency  represented  by  fuel  consumed. 
All  of  thee®  measures  were  considered  and  recorded  as  of  special  interest  $  how¬ 
ever,  again  as  a  matter  of  economy  and  convenience,  a  still  sore  restricted  set 
was  need  in  tho  following  analysis.  Thus,  the  neasures  used  in  this  analysis 
are  simply  terminal  displacement  and  altitude  and  oroosrang©  errors  (all  in 
feet).  Tho  altitude  and  oroaarsngo  measures  used  were  again  the  estimation 
(baaed  on  the  respective  cean  and  standard  deviation  of  error)  of  the  maximum 
error  occurring  with  95$  frequency,  as  for  pitch  error  in  the  orbital  insertions. 
The  criteria  sot  wore  50 » 000  feet  displacement  and  8,579*6  foot  in  both  altitude 
and  crossrange  error  (baaed  on  a  mean  of  2,000  and  a  standard  deviation  of  4,000 
foot).  Fuel  consumption  was  not  considered  because,  although  quite  possibly 
useful  in  differentiating  acong  different  levels  of  acceptable  performance,  it 
is  of  slight  relevance  to  nisaion  effectiveness  in  this  phase. 
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Bren  though,  the  number  of  measures  was  severely  limited,  in  the  fashion 
described,  there  still  remain  22  measures,  distributed  over  9  mission  phases, 
te  represent  each  pilot's  flight  control  performance  in  the  test  mission* 

These  are  presumed  to  effectively  represent  performance  aspects  which  are  both 
orltical  to  mission  success  and  sensitive  to  differences  in  performance  within 
and  among  individuals.  These  measures  are  summarised  (with  abbreviations) 
and  the  associated  criteria  are  given  in  table  V. 


i^lytioKethMPlsgy 

That  there  were  so  many  measures  of  pilot  performance  raised  significant 
problems  in  analytic  methodology.  It  is  conventional  in  psychological  research 
to  limit  the  measures  of  performance  (the  dependant  variable)  to  one  or  a  few 
measures,  taken  repeatedly.  This  arrangement  permits  a  determination  of  perform¬ 
ance  differences  directly  in  terms  of  the  measures  used  and  any  significant 
variation  in  performance  and  performance  differences  may  readily  be  attributable 
to  experimental  manipulations  in  the  variable  under  study*  But,  aa  applied  to 
this  circumstance,  such  a  method  would  lead  to,  again,  a  total  of  22  outcomes 
for  each  individual  and  comparison  of  interest— certainly  an  unmanageable  number 
to  interpret  affectively  in  a  direct  manner.  Furthermore,  the  22  measures  make 
reference  to  almost  as  many  different  ranges  of  variation  in  magnitude  and  5 
different  unite  of  measure  and  so  cannot  be  directly  combined  without  undeeired 
bias.  Nevertheless,  it  was  imperative,  aa  a  study  goal,  4  hat  the  data  bo  inter¬ 
preted  in  such  a  way  aa  to  permit  statements  concerning  overall  phase  and,  in 
turn,  mission  effectiveness — not  just  a  collection  of  statements  concerning 
performance  in  individual  parameters.  Also,  it  seamed  very  desirable  to  express 
these  overall  statements  in  terms  of  performance  reliabilities  (probabilities) 
rather  than  in  terms  of  arbitrary  units  of  measure.  Expression  in  terms  of 
reliability,  or  the  language  of  design  effectiveness,  would  enhance  the  caaning- 
fulr.es s  of  the  results  to  s  broader  readership  and  sake  ihm  more  directly 
applicable  to  design  problems. 

Therefore,  rather  than  to  limit  the  treatment  of  tho  recorded  data  to  a 
simple  transformation  in.~  2-scores  (i.e. ,  interpreting  the  raw  data  in  teres 
of  a  normal  density  function  having  the  same  estimated  parameter  values),  which 
is  a  common  method  for  reducing  multiple  measures  to  a  cossaon  basis  for  combina¬ 
tion,  a  novel  method  was  developed  and  employed.  Essentially,  this  involved 
the  interpretation  of  performance  on  each  parameter  at  any  stage  of  the  study 
(training,  teat  or  retort)  as  an  estimated  probability  of  performing  a4,  a 
specified  criterion  value.  These  probabilities  or  reliabilities  were  then  com¬ 
bined  in  each  of  two  ways  to  arrive  at  somewhat  different  kinds  of  statements, 
all  with  unique  and  useful  meaning,  concerning  performance  reliability  in  the 
various  phases  and  over  the  whole  mission.  Thus,  aa  a  result,  it  is  possible 
to  state,  alternatively,  the  following  about  perfonsance  on  any  given  occasions 

(l)  the  probability  of  mooting  criterion  performance  in  a  measured 
parameter  of  a  given  mission  phase}  defined  a a 

p -£jl 

m 
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(2)  the  probability  of  masting  criterion  performance  in  all 
measured  parameters  of  a  given  mission  phase,  defined  as 


(3) 


P  -Tp 

the  probability  of  meeting  criterion  performance  in  a 
measured  parameter  of  the  ©owlets  mission;  defined  as 


P 


t.E.  P 


(4) 


(5) 


the  probability  of  Easting  criterion  performance  in  all 
measured  parameters  in  a  phase  of  the  complete  mission 5 
defined  as 

l,  TFJfc 

j-iir. 

k 

the  probability  of  meeting  criterion  performance  in  all 
measured  parameters  of  all  phases  of  the  complete  mission! 
defined  as 

.  k  »k 

p-TJIJP 


In  each  of  the  defining  expressions  m  is  the  number  of  measures  in  a  phase 
and  k  is  the  number  of  phases  in  the  mission,1  Taken  together  these  five  indi¬ 
cants  represent  a  hierarchy  of  inclusivensss.  The  first,  p,  represents  merely 
the  average  probability  of  succiass  in  a  given  parameter,  as  estimated  on  the 
basis  of  a  given  phase j  whereas  the  last,  p,  represents  the  probability  of 
successfully  performing  the  whole  mission  (i.e.,  not  exoeeding  the  criterion 
for  any  parameter). 


Seen  in  another  way,  of  course,  the  indioant  p  also  represents  the  result¬ 
ant  of  a  particular  modal  of  pilot  flight  control  within  the  simulated  system, 

A  relevant  question  which  may  be  raised  then  is  whether  the  simple  modal  expressed 
by  relationship  5  is  appropriate  one  for  the  present  purpose.  While  not 
without  problems,  as  noted  later,  this  modsl  did  sees  to  be  the  bast  choice 
within  the  limits  of  present  methodology.  In  fact,  if  the  need  is  stringently 
expressed  it  may  bo  considered  the  only  ohoioe  possible.  A  general  discussion 
of  the  methodological  problem  of  modeling  (or  measuring)  complex  task  perform¬ 
ance  and  this  approach  to  it  is  planned  for  early  publication.  Appendix  II 
includes  a  description  of  the  analytic  details  applicable  to  this  study. 


Finally,  besides  the  problem  of  handling  the  different  measures  obtained 
in  auoh  a  way  as  to  arrive  at  useful  statements  concerning  phase  and  mission 
reliabilities,  there  is  yet  another  analytic  problem  posed  by  those  data.  It 
is  simply  that,  for  any  performance  comparison  of  interest,  except  certain 
comparisons  of  performance  <~u  the  part  of  the  same  individual,  there  is  only 
one  unique  measure  available.  Thus,  with  respect  to  retention  interval  eaoh 


For  readers  unfamiliar  with  the  other  symbols,  £  stands  for  "the  sum  of**  the 
item  series  indicated  and  T  stands  for  "the  joint  product  of"  the  item  series 
indicated. 
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(2)  t'm  probability  of  meeting  criterion  performance  in  all 
measured  parameters  of  a  given  mission  phase*  defined  as 


(3)  the  probability  of  easting  criterion  performance  in  a 
measured  parameter  of  th®  complete  sissionj  defined  as 

p  *  P 
I?  »"k 


(4)  the  probability  of  Besting  criterion  performance  in  all 
measured  parameters  in  a  phage  of  the  complete  mission* 
defined  as 

I- 

k 


(5)  the  probability  of  meeting  orit orion  performance  in  all 
measured  paraasstere  of  all  phases  of  the  complete  mission* 
defined  as 

,  k  Mk 

P  -  V  tf  p 


In  each  of  the  defining  expressions  a  is  the  number  of  measures  in  a  phase 
and  k  is  the  number  cf  phases  in  the  mission.^  Taken  together  these  five  indi¬ 
cants  represent  a  hierarchy  of  inclusivoneoa.  The  first,  p,  represents  merely 
the  average  probability  of  sucoosa  in  a  given  parameter,  as  estimated  on  the 
basis  of  a  given  phase *  whereas  the  last,  p,  represents  the  probability  of 
successfully  performing  the  whole  mission  (i.ee,  not  exceeding  the  criterion 
for  any  parameter). 

Seen  in  another  way,  of  course,  the  indicant  p  also  represents  the  result¬ 
ant  of  a  particular  modal  of  pilot  flight  control  within  the  simulated  system. 

A  relevant  question  which  may  be  raised  then  is  whether  the  simple  modal  expressed 
by  relationship  l>  is  an  appropriate  one  for  the  present  purpose.  Khila  not 
without  problems,  as  noted  later,  this  model  did  seem  to  be  the  best  choice 
within  the  limits  of  present  methodology.  In  fact,  if  the  need  is  stringently 
expressed  it  may  be  considered  the  only  choice  possible.  A  general  discussion 
of  the  methodological  problem  of  modeling  (or  measuring)  complex  task  perform¬ 
ance  and  this  approach  to  it  is  planned  for  early  publication.  Appendix  II 
includes  a  description  of  the  analytic  details  applicable  to  this  study. 

Finally,  besides  the  problem  of  handling  the  different  measures  obtained 
in  such  a  way  as  to  arrive  at  useful  statements  concerning  phase  and  mission 
reliabilities,  there  is  yet  another  analytic  problem  posed  by  these  data.  It 
is  simply  that,  for  any  performance  comparison  of  interest,  except  certain 
comparisons  of  performance  on  the  part  of  the  same  individual,  there  is  only 
one  unique  measure  available.  Thus,  with  respect  to  retention  interval  each 


1  For  readers  unfamiliar  with  the  other  symbols,  2  stands  for  "the  sum  of"  the 
item  series  indicated  and  T  stands  for  "the  joint  product  of1'  the  item  3©ries 
indicated. 
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mas 8  7 


treasures  Used  in  Analysis  of  Flight  Control 


‘Traaslumar  Insertion  (TLl)  ® 

voloolty  cutoff  error  (V)  10  fpa 

pitch  control  error  (P)  0*2645  deg 

Transposition  (TE2J) 

displacement  rat©  (JB.)  1  fpa 

impact  rat©  (SS)  1  fps 

fual  consumed  (?)  10  slags 

Lunar  Orbit  Insertion  (LOI) 

vslocity  outoff  error  (?)  10  fpo 

pitch  control  error  (?)  0*2645  dag 

Separation  and  Bsorbit  (SBO) 

velooiiy  cutoff  error  (V)  2  fps 

yaw  control  error  (Y)  1  dag 

pitoh  control  error  (?)  2  dog 


Brake  and  Hover  (BH) 


displacement  (or  range  error)  (?) 
displacement  rate  (®) 
ispaot  rate  (XP.) 
percentage  fuel  consumed  (?) 

200  ft 
10  fpS 
10  fps 

95  < 

Lunar  Aaeoni  (LA) 

volooity  cutoff  error  (7) 

10  fps 

Hondozvous  (Bend) 

percentage  fual  consumed  (?) 

yy  7° 

Booking  (Bok) 

displacement  (?) 
displacoasant  rat®  (IS) 
impact  rate  (XE) 

1  ft 

o»5  fps 
9*1  fpo 

Earth  Entry  (S3) 

diesplaoenent  (or  rang®  error)  (?) 
altitude  error  (A.) 
orosaraage  error  (o) 

50,000  ft 
8,579.6  ft 
8,579.6  ft 

*  Also  applicable  to  Transoaxth  Insertions  (TBl)  porfoxsod  by  0-4  and 
0-9  in  retraining* 


of  the  or^tfs  ■sas  tasted  at  a  sos-svk&t  different  internal  and  bo  may  not  {eoseept 
possibly  for  0-8  and  0-9)  be  dofen&ibly  combined.  Esther  evident  differences 
in  the  adequacy  of  original  training  in  tbs  two  phases  of  the  study  (as  will 
ba  shown)  argue  against  combining  0-S  and  0“9,  even,  if  &*  and  $~v3ek  intervals 
are  otherwise  considered  the  same  for  th©  purpose  of  this  study. 

Furthermore,  within  each  group  th®  arswsssbers  «ars  tested  in  £  unique 
order  on  the  supposition  (bom©  out  V  the  data)  that  those  tested  second  and 
third  might  benefit  from  participating  in  the  preceding  tests,  even  though  not 
acting  as  pilot.  In  addition,  as  is  also  suggested  by  the  data  on  performance 
at  the  end  of  training,  each  pilot  case  to  the  test  having  previously  acquired 
a  unique  level  of  skill  on  the  several  mission  tasks  not  necessarily  paralleling 
the  level  of  skill  achieved  by  any  other  cue.  Tot,  it  is  known  that  previous 
level  of  skill  & ay  strongly  influence  aacunt  of  measured  skill  retention.  There¬ 
fore,  the  performances  of  the  several  pilots  within  a  given  crew  may  not  be 
combined  for  cross-comparison  purposes  and  used  for  obtaining  error  estimates 
basic  to  testing  the  significance  of  differences  without  incurring  the  risk  of 
selective  bias  resulting  from  the  interactions  operating. 

The  net  consequence  of  these  facts  la  that  the  study  data  must  be  viewed 
and  the  results  consistently  interpreted  with  reference  to  the  rather  complex 
data  structure  illustrated  in  table  VI.  In  the  presentation  of  results  to 


TABLE  VI 

Structure  of  the  Iteta 


Crew/Betention  Stages  of  Experiment- 

Interval  Training  Test  Set raining 

0-4  ”  Pilot/ Order  1 

2 

3 

0-9  -  Pilot/Order  1 

2 

3 

0-8  -  Pilot/  Order  1 

2 

3 

0-13  -  Pilot/Order  1 

2 

3 
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follow  t ha  value®  obtained  for  tbs  several  orewssabare  of  a  given  ores  have 
bssst  (soabinsdj  on.  occasion*  Bit,  the  dubiety  of  this  procedure,  ia  view  of 
tbs  previous,  mould  ha  constantly  bom©  in  Bind*  Conventional  tests  of 
significance  war®  not  considered  appropriate  and  the  interpretation  of  the 
results  m&%  rest  upon  tho  observed  internal  and  logical  consistency  of  the 
data*  The  situation  is  frenldy  dsscrfbable,  frcsa  a  statistical  point  of  view, 
as  a  g©ro  degress  of  freedoms  case*  Fevertheloss,  the  nature  of  the  data  permits 
the  drawing  of  sera©  useful  conclusions* 
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SECTION  in 


sesoups 


In  keeping  with  the  previous  statement  on  methodological  considerations, 
the  main  results  of  the  study  pertinent  to  flight  control  are  presented  in  this 
section  in  tarns  of  probabilities  (or  reliabil it ies )•— first  with  reference  to 
skill  at  the  conclusion  of  training,  next  with  reference  to  retention  teat 
performance,  and  finally  with  reference  to  retraining  performance.  In  all  oases 
the  probabilities  reported  are  rounded  to  the  nearest  *001  with  the  result  that 
a  probability  of  ,0005  or  less  is  reported  as  <,001  and  treated  as  zero  in  any 
calculations  and  a  probability  of  .9995  or  greater  is  reported  as  >,999»  or 
simply  indicated  by  a  dash  in  tables,  and  treated  as  one  in  any  calculations. 

I5y  way  of  limiting  the  amount  of  tabular  material  to  that  most  immediately  use¬ 
ful,  the  basic  values  presented  are  the  probabilities  per  phase  of  success  in 
any  parameter  (p)  or  in  all  parameters  of  the  phase  (p).  These  may  be  thought 
of,  respectively,  as  the  average  probability  and  the  joint  probability  per  phase. 
More  inclusive  indicants  for  individual  pilots  and  crews  are  given  also.  How¬ 
ever,  the  elemental  probabilities  for  each  measured  parameter  (of  the  22  described 
earlier),  from  which  p,  p  and  the  still  more  inclusive  indicants  are  derived, 
may  be  found  in  the  tables  of  appendix  III  for  reference. 


The  first  question  which  must  appropriately  be  asked  of  the  study  data 
is  naturally,  Ehat  level  of  skill  was  achieved  by  the  test  personnel  prior  to 
the  retention  period  and  subsequent  test?  Research  on  skill  retention  baa 
frequently  demonstrated  that  level  of  prior  learning  strongly  influences 
measured  skill  retention.  Therefore,  it  is  important  to  assess  the  relative 
level  of  the  several  individuals  and  crews  involved  in  this  test  in  order  to 
validly  interpret  their  test  performance.  Otherwise,  for  example,  differences 
or  laok  of  differences  easily  attributable  to  differences  in  learning  might  be 
falsely  ascribed  to  differences  in  duration  of  retention.  In  this  case  assess¬ 
ment  of  prior  levels  seems  especially  important  because  the  impressions  of  study 
personnel  as  well  as  the  relative  numbers  of  recorded  training  experiences  in 
the  various  mission  phases  clearly  suggest  that  the  training  received  by  0-8 
and  0-13  was  superior.  Furthermore,  the  desire  to  generalise  the  study  find¬ 
ings  to  planning  for  apace  operations  implies  that  the  prior  skill  achievement 
of  the  test  personnel  should  be  demonstrably  high.  For,  it  may  be  assumed  that 
crews  of  space  systems  will  continue  (at  least  for  some  time  to  come)  to  be 
trained  to  the  near  limits  of  their  skill  potential. 

Mobility  pejc  _ggBQiMtJ.S6l  CyltPjlft.  One  approach  to  assessing  prior 
skill  achievement  is  to  simply  compare  the  best  performance  of  the  pilots, 
presumably  occurring  at  the  end  of  their  training,  with  the  hypothetical  system 
criterion  for  each  of  the  several  measures  of  interest.  If  the  criteria  selected 
do  represent  reasonable  or  typical  mission  requirements,  as  is  supposed,  then 
suoh  a  comparison  will  show  the  oapability  of  the  pilots  to  perform  the  mission. 
Moreover,  since  the  performance  of  eaoh  individual  and  crew  would  be  asseosed 
relative  to  a  common  reference,  any  differences  in  achievement  should  beocsse 
evident. 
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The  results  of  such  a  comparison  ar®  summarised  in  table  VII.  In  this 
table  are  listed  for  each  pilot  the  phaae-by-phasn  reliabilities  at  tk©  end 
of  training,  as  estimated  fro®  the  last  four  training  trials  (with  the  Past 
Time  and  Eoal  Tine  Mission  excluded) .  (See  appendix  II  for  details  concerning 
computation  of  these  values. )  In  each  case  the  two  primary  values,  ]5  and  p, 
art?  given.  These  primary  values  are  then  taks.n  cumulatively  aoross  phases  to 
obtain  the  values  f  and  p  and  jointly  to  obtain  i>  for  the  complete  mission  for 
each  pilot.  Finally,  the  p  mid  t>  values  for  phases  and  the  f ,  5  and  f>  values 
for  the  mission  for  the  three  individual  pilots  within  a  crew  are  taken  cumula¬ 
tively  to  obtain  the  values  of  f  and  $  for  phases  and  f,  p  and  1  for  the  mission 
for  each  crew.  Logically,  these  last  five  reliabilities  indicating  cyew  perform¬ 
ance  merely  state  the  most  likely  reliability  to  ba  expected,  of  the  nature 
otherwise  indicated,  if  a  member  of  that  particular  crew  performs  the  particular 
phase  or  the  mission  (when  performance  by  the  members  is  equally  likely).  This 
method  of  tabulation  will  be  repeated  in  other  tables  aa  well® 

Even  rather  cursory  examination  of  table  VII  suggests  that  there  were 
differences  in  skill  at  the  end  of  training  which  are  worthy  of  note.  First, 
for  ail  phasoB  the  value  of  p  does  not  exceed  the  value  of  p  and  generally 
(except  where  only  one  measure  is  involved)  it  is  smaller.  This  is  to  ba 
expected  from  the  difference  in  algebraic  manipulation  defining  each.  Hence, 
evidently  and  by  definition  p  is  the  more  sensitive  indicant,  such  that  the 
lowest  value  of  p  obtained  for  any  pilot  in  any  phase  is  .002  (P-41  in  EB) 
whereas  the  lowest  value  of  p  obtained  is  .158  (0-91  in  LA).  Both  indicants 
range  upward  in  a  number  of  instances  to  >.999* 

Second,  evidently  there  was  considerable  variation  in  the  pilots1 
capability  to  perform  the  various  mission  phases.  Thus,  for  example,  p  for 
P-41  varied  from  a  high  of  >.999  in  the  LOI  to  a  low  of  .259  for  KE,  p  ranging 
from  >.999  to  .002  for  the  same  phases.  Similarly  P-91  ranged  from  .816  to 
.158  in  p  and  .632  to  .158  in  f>.  However,  in  contrast  P-8l  and  P-131  showed 
much  less  individual  variation.  There  is  a  suggestion  of  similarity  among 
pilots  in  relati”e  effectiveness  on  the  several  phases,  especially  among  those 
of  0-4  and  0-9  where  EE  is  generally  performed  less  well,  but  there  seems  to 
be  no  such  entirely  consistent  pattern*  Individual  differences  as  well  as 
selective  differences  in  training  might  easily  have  contributed  to  this  restilt. 

Third,  it  is  evident  also  that  so  far  as  overall  mission  capability  at 
the  end  of  training  is  concerned  the  pilots  differed  considerably.  Thus,  p 
ranged  from  .587  to  .991  ( f>  from  .312  to  .978  and  p  from  <.001  to  .747)  within 
this  sample  of  12.  Once  again,  of  course,  both  differences  in  training  a a  well 
as  in  sld.ll  potential  may  have  contributed  to  this  result. 

Finally,  the  mission  capabilities  of  the  pilots  comprising  the  several 
crews  olearly  suggests  that  pilots  of  0-4  and  0-9  were  much  less  skilled  at 
the  end  of  training  than  ware  pilots  of  0-8  and  0-13.  This  difference  is 
olearly  confirmed  by  tLe  reliability  estimates  for  the  phases  and  the  mission 
for  each  whole  crew.  Thus,  the  mission  ^liabilities,  p,  for  0-4  and  0-9  were 
.726  and  .648  as  contrasted  with  .984  rnd  .953  obtained  for  0-8  and  0-13  and 
comparable  differences  exist  for  p  and  p  values  aa  well.  An  illustration  of 
these  differences  between  0*4  and  0**9  when  oombinod  and  0-8  and  0*13  when 
combined  is  given  in  figure  7  which  depicts  overall  f  and  p  values.  When  the 
information  on  the  rendezvous  is  eliminated  in  oompirt ing  the  values  for  0-8 
and  0-13  to  provide  a  common  8~phase  basis  for  comparison  the  differences  are 
even  greater. 
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TABLE  VII 


Reliability  Per  Criteria  at  the  End  o.  Training 


Crow/ 

Pilot  TL1 

TIE?  L01 

SID 

Phase 

B3  LA 

Rend 

Bok 

BE 

Kission  v 

5,5  * 

0“4  1  (?)  .892 

.745  ~T 

.928 

.450  .358 

ND 

.726 

.259 

.670 

(?)  .785 

.234  — 

.785 

.030  .358 

.228 

,002 

.428  <.001 

2  .957 

.547  .798 

.827 

.999  .  345 

KB 

.289 

.535 

.662 

.9H 

.054  .  600 

.480 

.997  .345 

.022 

.129 

.443  <.001 

3  .928 

.668  — 

.864 

.995  — 

KB 

.751 

.55A 

.845 

.  860 

.016  — 

.593 

.981  — 

.392 

.107 

.619  <.001 

all  (p)v  .926 

.653  .933 

.873 

.815  .568 

KB 

.589 

.449 

.726 

IP)  .853 

.101  .867 

.619 

.669  .568 

.214 

.079 

.497  <.001 

0~9  1  (?)  .816 

.614  .761 

.472* 

.618  .158 

SB 

.637 

.619 

.58? 

(?)  .632 

.170  .545 

.159  .158 

.186 

.176 

.312  <.001 

2  .534 

.917  .996 

.339* 

.813  KB 

KB 

.723 

.654 

.711 

.276 

.752  .992 

•395 

.220 

.256 

.461  .002 

3  .750 

.691  — 

.863* 

.93?  .170 

1TB 

.506 

.243 

,646 

.500 

.287  — 

.755  .170 

.116 

.003 

.462  <.001 

all  (?)  .700 

.741  .919 

.560* 

.789  .164 

ND 

.622 

.505 

.648 

(?)  .469 

.403  .846 

.436  .164 

.174 

.145 

.412  ,001 

0-8  1  (?)  .990 

.919 

« _ 

.990 

(?)  .979 

— 

.757 

—  ~ 

— 

— — 

•— 

.971  .741 

2  — 

— »  - 

.953 

—  .806 

.996 

— 

.973 

~ - 

.999  — 

.860 

.999  .80 6 

— 

.988 

.961  .683 

3  .954 

—  .yyc 

AAA 

oyyy 

— —  - 

— » 

<VtO 

•  7  I4- 

.991 

.908 

—  .983 

.998 

— 

.917 

— 

.978  .81? 

all  (?)  .981 

—  .99? 

.957 

~  .935 

— 

.989 

_ 

>984 

(?)  .962 

—  .994 

.872 

-  - 

“ 

.968 

— 

.970  .747 

0-13  1  (?) 

|  11  — n 

—  .999 

.932 

.980 

.999 

.789 

_ 

— — 

.967 

)  - - 

—  .999 

.804 

.920 

.999 

.785 

— 

— 1 - 

.946 

.582 

2 

— 

-  ~ 

.739 

— 

.999 

.978 

.939 

.998 

.961 

“ —  — 

.370 

.999 

.978 

.319 

.994 

•  907 

.294 

3 

.903 

— 

.995 

.998 

.557 

.997 

.942 

— » 

.932 

.806 

.999  __ 

.984 

.993- 

.557 

.997 

.827 

— — 

.907 

.361 

all  (p) 

1  .968 

— -  - 

.889 

.993 

.852 

.921 

.960 

.999 

.953 

(?) 

1  .935 

- -  — 

.719 

.971 

.852 

.921 

.882 

.998 

.920 

.412 

t  dash  indicates  value  >.9995  v 
*  based  on  reduced  data 

ND  data  we, -  unavailable 
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probabilities  for  "all”  pilots  ar© 
individual  values  taken  cueulatively 
(i.e.t  meanc  of  individual  valuse) 


Those  apparent  and  large  differences  between  the  crews  of  the  two  test 
phases  and  the  nature  of  the  reliabilities  obtained,  when  viewed  in  the  context 
of  the  preceding  observations  conoaming  level  of  learning,  cast  serious  doubt 
upon  the  value  of  the  data  from  the  first  two  crews  tested  for  supporting  any 
useful  generalizations  on  retention.  Certainly,  any  generalizations  concerning 
the  performance  of  0-4  and  0-9  oust  be  duly  related  to  the  evidently  lees  than 
complete  training  received  by  these  crews,  for  there  is  no  evident  basis  for 
contending  that  they  were  inherently  less  capable. 

Capability  Relative  to  Skill  Potential,  However,  that  0-8  and  0-13  were 
by  far  the  more  capable  crews  and  reached  rat*,  ar  high  levels  of  reliability 
according  to  arbitrary  criteria  does  not  establish  that  even  they  had  reached 
anywhere  near  the  limits  of  their  skill  potential.  Instead,  to  determine  their 
actual  level  of  skill  relative  to  their  potential  what  is  needed  is  some  way 
of  determining  when  improvement  in  measured  skill  has  stopped  and/or  a  method 
o?  specifying  levels  of  learning  (i.e.,  training)  short  of  or  beyond  that  level. 
It  has  often,  bo  m  demonstrated  that  practice  beyond  the  point  of  further  measured 
improvement  sti*l  brings  continued  learning  (technically,  called  over-learning) 
as  evidenced  by  improved  retention.  But,  the  traditional  means  for  attempted 
satisfaction  of  this  need  aru  neither  standardized  nor  adequate.  Thus,  aoramonly 
the  assessment  of  asymptotic  performance  rests  upon  a  personal  judgment  o?  the 
investigator  as  to  whether  the  mean  performance  taken  over  successive  blocks 
of  trials  is  reasonably  stationary.  Levels  of  learning  (or  training)  are  usually 
specified  in  terms  of  actual  practice  time  or  practice  trials,  relative  to 
practioe  or  trials  required  for  asymptotic  performance  level,  as  estimated  from 
the  same  data  (when  the  asymptote  was  reached)  or  like  data  (when  the  asymptote 
was  not  reached,).  To  specify  asymptotic  performance  by  means  of  a  personal 
judgment  is,  of  course,  to  invite  inaccuracy  in  the  specification.  To  specify 
level  of  training  short  of  the  asymptote  on  a  personalized  crude  extrapolation 
from  typical  data  and  the  terminal  rat6  observed  is  to  court  still  greater 
inaccuracies.  Furthermore,  to  specify  the  asymptote  or  the  level  in  terms  of 
cne  mean  performance  is  to  discount  the  variation  m  performance  from  instance 
to  instance  in  the  face  of  the  commonly  held  belief  that  good  performance  is 
not  only  typically  good,  but  reliable  as  well.  On  the  other  hand,  if  the 
performance  changes  indicated  by  even  the  means  of  successive  performance 
samples  are  considerable  and  consistently  in  the  direction  of  improvement  it 
may  certainly  be  concluded  that  skill  has  not  yet  reached  peak  levels. 

The  data  on  training  and  retraining  performance  in  •she  present  study 
were  thus  examined  with  an  eye  to  roughly  estimating  now  near  the  test  personnel 
appro&ohed  their  sdll  limits  at  the  end  of  training.  However,  in  making  this 
analysis  and  in  keeping  with  the  study  goals  primary  attention  was  given  to  the 
variation  it  estimated  performance  level  for  successive  performance  samples 
which,  from  the  data,  were  estimated  as  achievable  with  a  high  degree  of  reli¬ 
ability.  Thus,  instead  of  considering  the  successive  levels  of  arithmetic 
"'ean  (or  presumably  most  typical  performance)  the  successive  levels  of  perform¬ 
ance  estimated  (from  four-ir-’al  blocks)  to  include  95^  of  performances  under 
exactly  Aha  same  circumstances  were  examined.  In  this  way  changes  in  vanabil- 
i'  -  as  well  aa  in  typical  performance  level  were  represented.  Although  95 v> 
admittedly  does  not  represent  an  extremely'  high  reliability'  it  does  represent 
an  effective  compromise  between  degree  of  certainty  and  increasing  errors  in 
estimating  the  associated  performance  level.  The  logic  for  this  approach  to 
performance  measurement  is  described  in  a  separate  article  by  Co+*eruan  i ’  ‘  7 
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Viewed  in  this  ’~>y  the  training  and  retraining  data  from  0-4  and  0-9 
simply  nor  firm  the  pi  ^.ously  mentioned  conclusion  about  their  end-of-H  raining 
skills.  There  is  great  variation  in  estimates  of  p  05  levels  for  successive 
four-trial  blocks  in  training  and  in  most  instances ’there  is  still  a  large 
trend  toward  improvement .  The  comparable  or  superior  levels  displayed  in 
re; raining,  which  will  be  presented  in  more  detail  later,  provide  further 
confirmation. 

However,  the  estimated  p^sj  levels  in  training  for  0-8  and  0~13  do  not 
permit  such  a  clear,  unambiguous  conclusion.  For  any  given  individual  the 
series  of  estimates  seems  to  have  generally  stabilized  for  some  parameters  and 
not  to  have  stabilized  for  others,  even  within  a  phase,  and  this  pattern  is  not 
consistent  among  individuals.  The  stability  noted  is  generally  to  be  found  at 
the  very  end  of  training  and  to  involve  only  a  limited  number  (2  or  3)  four- 
trial  blocks.  Examples  of  these  training  functions  (and  retraining,  as  well) 
are  given  in  appendix  IV  in  which  the  data  from  P-131 .are  dapiotad  graphically. 

In  these  graphs  the  estimated  P.95  level  is  coded  by  X  to  distinguish  it  from 
the  mean,  or  X,  and  the  recorded  value  on  a  given  trial,  or  X,  in  traditional 
coding.  A  reasonable  interpretation  of  these  results  wold  b8  simply  that  by 
the  end  of  training  the  several  pilots  involved  neared  in  varying  degrees  on 
the  several  parameters,  bit  had  not  quite  reached  (with  the  exception  of  certain 
parameters)  the  limits  of  their  skill  potential.  This  interpretation  also  is 
supported  ty  the  retraining  results  to  be  presented  in  detail  later. 

Further  confirmation  of  this  evaluation  of  the  end-of-training  skill  of 
0-8  and  0-13  may  be  derived  (depending  on  personal  judgment)  from  the  limited 
application  of  a  newly-developed  decision  rule  to  their  training  data.  This 
training  decision  rule  provided  a  means  for  more  precisely  determining  when 
asymptotic  skill  level  was  achieved  for  the  purpose  of  distributing  the  avail¬ 
able  training  time  for  greatest  overall  training  effectiveness.  However,  when 
applied  to  the  data  in  question  in  only  a  few  cases  did  the  outcome  result  in 
a  finding  of  skill  stabilization  (i.o.,  peak  skill  attainment).  It  seemed 
apparent  that  available  training  tims  would  not  permit  skill  stabilization,  at 
least  as  indicated  by  the  decision  rule  used.  Since  additional  calculations 
not  considered  directly  relevant  to  aii&lysia  with  reference  to  system  criteria 
were  involved,  with  attendant  coats,  the  study  manager  abolished  the  require¬ 
ment  for  its  application.  The  decision  rule  concept  is  described  in  a  general 
way  jiswenere  (Cottemar.,  1967)  but  it  has  not  been  elaborated  and  properly 
tert^d.  Therefore,  being  as  yet  unproven  the  rale  may  be  questioned  as  to 
vnliiitv  and,  1  r.  the  form  used,  as  to  excessive  stringency.  For  the  moment, 
then,  t ’.e  in  .erpretation  of  the  outcomes  obtained  with  it  seem3  necessarily  a 
matter  of  personal  judgment. 

Nevertheless,  whacever  the  validity  of  the  decision  rule,  because  stability 
m  estimated  5,95  performance  level  would  certainly  constitute  a  severe  require¬ 
ment  by  usual  standards  it  seems  fair  to  view  C-8  and  0-13  as  having  reared,  but 
not  quits  Touched  in  training  the  limits  of  their  potential  skill.  Such,  although 
never  precisely  determined,  is  not  unlikely  the  result  of  most  operational  train¬ 
ing  programs.  Accordingly,  the  data  of  0-8  and  0-13  are  taken  provisionally  as 
a  reasonable  basis  for  generalizations  concoming  skill  retention  and,  in  the 
following  description  of  mission  test  and  retraining  performance,  they  are  con¬ 
sidered  of  primary  significance.  The  parallel  analyses  for  C-4  and  C-f  are 
presented,  but  they  are  considered  of  leaser  ■'■alue. 


37 


Reliability  per  Hypothetical  Criteria*  Bo*  wail  than  did  th©  craws  per¬ 
form  in  the  tost  mission  of  main  concern?  Since  end-o f-t rain ing  performance 
has  been  expressed  as  sets  of  estimated  reliabilities  with  reference  to  22 
flight  control  criteria,  an  evident  approach  to  answering  this  question  is  to 
compare  with  them  similar  estimates  based  on  performance  in  the  test  mission. 

The  performance  effects  of  the  retention  periods  should  then  be  reflected  in 
differences  between  end-of-training  and  test  mission  reliability  estimates. 
Hence,  the  p  and  £  values  per  phase  for  each  pilot  were  computed,  as  before, 
and  the  more  inclusive  f,  f>  and  j>  values  for  each  individual  and  the  parallel 
cumulative  probabilities  for  crews  determined-.^  All  these  values  are  presented 
in  table  VIII,  which  ia  organised  in  the  same  fashion  and  may  be  compared 
directly  with  table  VII  concerning  end-of-t raining  performance. 

First,  taking  the  values  of  table  VIII  strictly  on  their  own  merits,  it 
is  evident  that  the  pilots  showed  considerable  individual  variation  in  capabil¬ 
ity  to  perform  the  several  mission  phases  in  the  test  mission  as  at  the  end  of 
training.  Among  the  commanders  (those  first  tested),  for  example,  p  for  P-Bl 
varied  from  >.999  in  a  number-  of  phases  to  .684  in  TL1,  and  p  from  >.999  in  the 
same  several  phases  to  .138  in  3 DO.  The  variation  in  phase  performance  of  P-131 
is  even  more  extreme,  ranging  from  >.999  to  .294  in  p  and  >.999  to  <.001  in  p. 
Such  variation  in  phase  performance  is  also  typical  of  all  the  pilots  of  0-4 
and  C~9  but  it  did  not  occur  in  the  performance  of  the  0-8  and  0-13  pilots  who 
were  tested  second  and  +hird.  The  more  consistent  and  superior  performance  of 
the  second  and  third  pilots  of  C-8  and  0-13  may  be  the  result  of  their  greater 
skills  (compared  with  those  of  0~4  and  O-9)  and  their  opportunity  to  observe 
the  first  tested  pilots  before  performing  the  test  mission  themselves.  Again, 
there  was  a  tendency  for  all  pilots,  especially  within  a  given  crew,  to  perform 
certain  phases  better  than  others,  but  the  pilots  were  not  entirely  consistent 
in  thia  "'aspect. 

In  fact,  when  the  phase  reliabilities  for  the  several  pilots  within  a 
given  crew  are  compared  there  is  considerable  agreement  on  some  and  consider¬ 
able  variation  on  others.  Almost  invariably  in  0-8  and  0-13,  the  performance 
of  the  first  tested  pilot  was  least  adequate,  suggesting  rather  definitely  the 
effects  of  order  of  test  among  those  about  equally  and  well  trained  individuals. 
That  similar  relations  are  not  as  evident  among  the  test  performances  of  0-4 
and  0-9  pilots  is  not  surprising  in  view  of  the  incompleteness  of  their  train¬ 
ing  and  greater  variation  in  their  terminal  skill. 


Because  no  direct  estimate  of  variability  could  be  made  from  the  single 
measure,  in  calculating  teat  mission  probabilities  the  variability  at  the  end 
of  training  was  used.  (See  appendix  II  for  details.)  In  general,  it  may  be 
expected  that  this  procedure  resulted  in  an  underestimate  or  overestimate  of 
test  mission  variability,  depending  on  whether  performance  in  the  test  mission 
was  better  or  worse  than  ©nd-of-t raining  performance.  Thus,  considering  the 
nature  of  these  data,  it  is  supposed  that  the  test  mission  reliabilities 
reported  are  usually  overestimates.  This  matter  is  more  fully  discussed  in 
the  following  section. 
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Ha I lability  P©r  Criteria  in  the  Test  Missiraa 


Crew/ 

Pilot  TL1 

ten  loi 

SIS 

Phase 

EH 

LA 

Bond 

Dok 

SS 

Mission  * 

t 

04  1  (?)  -~f 

.752  — 

.531 

.350 

.635 

HD 

.187 

.737 

.649 

h)  — 

.255  — 

<.001 

.099 

.635 

KD 

<.001 

.305 

.412  <.001 

2  .999 

.828  .919 

.530 

.998 

.655 

ND 

.654 

.418 

.750 

.998 

.526  .838  <.001 

.994 

.655 

ND 

.277 

.024 

.539  <.001 

3  .992 

.705  ~ 

.826 

.861 

ND 

.762 

.380 

.816 

.983 

.116  — 

.497 

.506 

— 

ND 

.412 

.005 

.565  <.001 

all  (?)*  .997 

.762  .973 

.629 

.736 

.763 

ND 

.534 

.512 

.738 

(?)  .994 

.299  .946 

.1 66 

.533 

.763 

ND 

.230 

.111 

.505  <.001 

09  1  (p)  .964 

.243  .946 

ND 

.250 

.583 

ND 

.640 

.740 

.624 

(?)  .928 

<.001  .893 

ND  • 

<.001 

.583 

ND 

.188 

.382 

.425  <.001 

2  .752 

.692  .958 

.517* 

.250 

ND 

ND 

.540 

.793 

.643 

.556 

.075  .916 

.517*<.001 

ND 

ND 

.092 

.462 

.374  <.001 

3  .898 

.830  — • 

.843* 

.635* 

.557 

ND 

.443 

.747* 

.744 

.797 

.536  — 

.843* 

.130* 

.557 

ND 

.080 

.521* 

.558  .001 

all  (?)  .871 

.588  .963 

.680 

.378 

.570 

ND 

.541 

.760 

.670 

(?)  .760 

.204  .936 

.680 

.043 

.570 

ND 

.120 

>455 

.452  <.001 

0-8  1  (?)  .684 

.713 

— 

„ 

•  999 

.981 

.931 

(?)  .367 

—  — 

.138 

— 

— 

— 

.998 

.942 

.827  .048 

2  .750 

~  - 

.980 

.968 

.966 

— 

— 

— 

.963 

.500 

-  — 

.939 

.871 

.966 

— • 

.920  .395 

3  .993 

—  .992 

.990 

-r— 

— 

.982 

-228 

.QQS 

.986 

—  .983 

.970 

— 

— 

- - 

.946 

.995 

.987  .885 

all  (?)  .809 

—  .997 

.894 

.989 

.989 

— 

.994 

.993 

.963 

(?)  .618 

—  .994 

.682 

.957 

.989 

— 

.981 

.979 

.911  .443 

013  1  (p)  — 

.960 

.294 

,903 

.936 

.899 

(?)  — 

-  - 

.880  <,001 

.903 

.936 

— — 

.858  <.001 

2  .996 

_  _ 

.893 

.964 

— 

.992 

•  997 

.982 

.993 

-  - 

.695 

' - 

.964 

— 

.975 

.992 

.958  .643 

3  .961 

.999  — 

.976 

.981 

.716 

.951 

— 

.954 

.922 

.996  — 

,928 

.925 

.716 

— 

.853 

— — 

.927  .481 

all  (?)  .986 

.943 

.758 

,861 

.979 

.981 

.999 

.945 

(?)  .972 

.909  — 

.834 

.642 

.861 

.979 

.943 

.997 

.914  .375 

t  dash  indicates  value  >.9995  v  probabilities  for  "all"  pilots  are 

*  based  on  reduced  data  individual  values  taken  cumulatively 

hP  data  wore  unavailable  (i.o.,  uoana  of  individual  values) 
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The  overall  mission  performance  of  the  pilots  as  indicated  by  B  clearly 
and  consistently  ^monstratee  these  effects  of  testing  order*  within  each  orew 
the  least  reliable  was  the  first  tested.  The  more  stringent  indicants,  ?  and  $, 
also  show  this  effect  to  the  extent  that  a  probability  of  sufficient  magnitude 
(that  la,  .000$)  for  discriminating  differences  waa  found.  Furthermore,  in 
general  the  performance  of  the  C-8  and  0-13  pilots  ms  considerably  superior 
to  that  of  the  0*4  and  0-9  pilots,  as  would  be  expected  from  the  differences 
in  skill  at  the  end  of  training  which  are  reflected  in  table  VII, 

However,  when  the  mission  reliabilities  in  table  VIII  for  individuals 
and  for  crews  are  contrasted  with  the  comparable  values  o?  table  VII  some  loss 
in  reliability  over  the  retention  interval  seems  evident.  For  example,  the 
estimated  reliabilities  for  F-131  decreased  from  .967  to  .899  in  from  .946 
to  .858  in  ]5  and  from  .582  to  <«0Q1  in  from  training  to  the  test  mission. 

Only  one  of  tho  four  pi.lots  first  tested  (P-9l)  did  not  show  an  overall  loss 
and  that  this  pilot  did  not  may  easily  be  considered  the  result  of  inadequate 
slri.ll  in  the  first  place  (p  of  .587  at  the  end  of  training).  Graphs  illustrat¬ 
ing  the  losses  in  reliability  indicated  by  test  performance  are  giver,  in  figure  8 
in  which  the  phase  reliabilities  (p)  of  P-81  and  P-131  in  training  and  in  the 
tost  mission  are  plotted.  As  crews,  both  C**8  and  0-13  show  overall  losses  even 
though  certain  individuals  showed  slight  gains.  That  C~4  and  0*9  did  not  show- 
overall  losses  may  again  bs  attributed  to  the  inadequacy  of  their  training. 
Because  the  pilots  of  those  crews  had  gained  less  skill  originally,  they  had 
relatively  less  skill  to  lose. 

These  relationships  between  overall  mission  performance  in  training  and 
test  are  more  conveniently  and  precisely  summarized  in  table  IX  which  shows 
both  the  absolute  amount  of  change  in  reliability  from  training  to  test  and  the 
percentage  of  end-of-t raining  reliability  that  change  represents.  Thus,  aooord- 
ing  to  table  IX  the  first  tested  pilots,  whose  performance  is  of  primary  interest 
because  it  is  uncontaminated  by  prior  participation,  nevorlost  more  than  .068 
or  7 fo  of  end-of-training  reliability  in  p.  Because  |  and  £  are  more  stringent 
indicants,  the  decrements  in  them  are  greater,  as  much  as  ,144  or  14*8^  and 
.562  or  100f$,  respectively.  What  these  figures  mean,  of  oourse,  is  that  while 
cumulative  (or  average)  parameter  reliabilities  held  up  rather  well  in  th.8  test 
mission  the  likelihood  of  mission  success  (as  defined  by  meeting  all  hypothetical 
criteria)  dropped  greatly.  This  degradation  in  t>  is  invariably  the  result  of 
degradation  in  only  one  or  a  few  parameters,  as  may  be  seen  more  directly  by 
comparing  tables  XXV  and  XXVI  of  appendix  III. 

The  shifts  from  training  to  test  in  parameter  (?)  and  phase  (?)  reli¬ 
abilities  for  the  wnole  mission  are  not  only  relatively  small  but  they  also  do 
not  seem  to  be  ordered  consistently  in  magnitude  according  to  retention  interval. 
It  would  be  expected  that  if  other  factors  are  equalized  the  longer  the  retention 
interval  tho  greater  the  loss  (or  lesser  the  gain)  in  reliability,  *mong  the 
first  tested  pilots  (the  test  01  whom  was  unoontaminated),  while  P-131  did  show 
the  expected  slightly  greater  loss  than  P-8l  and  P-8l  greater  loss  than  P-41, 
the  gain  of  P-91  is  completely  out  of  order.  Thu«,  instead  of  F~41< P-8l£ F-91 
<P-131  the  order  is  P~91< P-41< ?-8l< P-131.  That  P-91  lost  nothing  (in  fact, 
gained)  in  teat  may  again  be  explained  on  the  grounds  of  his  comparative  lack 
of  skill  at  the  end  of  training*  that  is,  he  bad  comparatively  little  to  lose. 
However,  even  granting  this  explanation,  because  the  differences  observed  are 
slight  it  can  hardly  be  concluded  that  the  leigih  of  the  retention  interval  had 
nmoh  effect.  There  appears  little  basis  in  these  results  for  arguing  that  a 
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Figuro  8,  Phase  Reliabilities  < 
Test  Mission,  ar.d  in 
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®iaug©  ia  Mission  Sol  lability  pay  Criteria 
fXC53  Training  to  Test  Mission  » 


Pilot 


Cre^Ksasuro 

1 

2 

3 

All  (sean) 

4  f 

-.021 

H.l) 

.088 

(lj*3) 

-.029 

(-3.4) 

.013 

(2.3) 

5 

-.016 

(“3.7) 

.096 

(21.7) 

7.054 

(-8.7) 

.009 

(3.1) 

* 

HM 

HM 

m 

HM 

9  P 

.037 

(6.3) 

-.068 
(~9. 6) 

.098 

(15.2) 

.022 

(4.0) 

.113 

(36.2) 

-.08? 

(-18.9) 

.096 

(20.8) 

.041 

(12.7) 

f> 

m 

-.002 

(-100.0) 

HM 

m 

8  f 

~.°59 

(-6.0) 

-.010 

(-1.0) 

.004 

(0.4) 

-.022 

(-2.2) 

5 

-.144 

(-14.8) 

-.041 

(-4.3) 

.009 

(0.9) 

-.059 

(-6.1) 

i 

-.693 

(-93.5) 

-.288 

(-42=2) 

.068 

/ 

-.304 

\  T-vy  / 

13  f 

-.068 

(-7.0) 

.021 

(2.2) 

.022 

(2.4) 

-.008 

(-0.8) 

I 

-.088 

(-9.3) 

.051 

(5.6) 

.020 

(2.2) 

-.006 

(-0.5) 

: 

P 

-.582 

(-100.0) 

.349 

(118.7) 

.120 

(33.2) 

-.036 

(17.3) 

*  percentage  gain  or  loos  indicated  in  parent ho cos 
EM  not  ssoningfa)  because  training  reliability  <.0005 
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13-week  retention  period  has  much  greater  significance  for  subsequent  perfona- 
anoe  than  a  4-  or  8-week  retention  period.  The  average  shifts  of  the  crews, 
being  ordered  0-9<0-4<013<0-8,  support  this  ease  view. 

However,  as  the  overall  reliability  shifts  of  table  VTi  for  pilots  within 
given  crews  are  compared  it  is  once  again  evident  that  excepting  P-91  those 
tested  first  tended  to  perform  less  well.  Certainly  this  is  true  of  0-8  and 
0-13  for  which  no  obvious  question  of  training  adequacy  can  be  raised.  That 
the  second  and  third  tested  individuals  performed  so  well,  relatively,  suggests 
a  considerable  training  (i.e.,  retraining)  value  accrued  to  them  merely  from 
observing  and/or  aiding  one  or  two  prior  performances. 

Capability  Relative  to  Previous  Level.  Thus,  to  the  extent  that  the 
hypothetical  criteria  are  representative  of  those  existing  in  real  space 
missions  and  the  analytic  methodology  employed  affords  valid  estimates  of 
reliability  a  number  of  useful  inferences  concerning  human  reliability  in  space 
operations  seem  possible  from  the  preceding  analysis.  These  inferences  will  be 
developed  more  explicitly  and  discussed,  as  warranted,  in  the  following  section 
of  the  report.  Meanwhile,  in  focusing  attention  in  the  analysis  so  lar  presented 
on  the  several  pilots1  and  crews'  capabilities  relative  to  criteria,  some  informa¬ 
tion  (as  it  happens  in  this  case)  has  been  lost  on  the  full  effects  of  the 
retention  period.  This  circumstance  results  from  the  fact  that  at  the  end  of 
training  the  pilots'  reliabilities  with  respect  to  many  of  the  measured  para¬ 
meters  far  exceeded  the  maximum  discriminable  value  (.9995)  adopted  as  practic¬ 
able  in  this  analysis.  Therefore,  an  individual  might  well  have  performed  less 
adequately  in  test  without  this  fact  being  reflected  in  the  calculated  probabil¬ 
ity.  That  would  always  be  true  when  capability  exceeds  the  maximum  discrimin¬ 
ate  reliability  relative  to  a  particular  measure.  In  effect,  then,  to  compute 
reliabilities  relative  to  criteria  in  these  circumstances  is  to  reduce  the  capa¬ 
bility  of  the  analysis  to  show  certain  changes  in  performance.  Nevertheless, 
in  this  study  and  in  many  others,  one  is  properly  interested  in  any  changes  in 
performance  which  occur — not  just  in  those  changes  which  have  obvious  and 
immediate  implications  for  operational  performance. 

Therefore,  as  a  means  of  assessing  more  fully  the  nature  of  the  perform¬ 
ance  in  the  test  mission,  contrasted  with  that  of  training,  an  additional 
analysis  of  the  test  data  was  performed.  In  this  analysis,  (again  in  keeping 
with  the  view  that  performance  excellency  is  indicated  by  the  level  that  can 
be  achieved  reliably,  not  just  typically;,  reference  is  made  to  the  estimated 
level  of  performance  which  the  individual  would  achieve  in  95$  of  his  perform¬ 
ances  under  like  circumstances  (i.e.,  his  p  05  level).  Thus,  with  respect  to 
each  flight  control  parameter  of  interest,  the  p  .r  level  of  performance  esti¬ 
mated  from  the  last  four  training  trials  was  usea  as  the  reference  against 
which  the  test  mission  performance  was  compared.  As  in  the  previous  analysis 
the  actual  performance  in  test  was  used  as  a  basis  for  estimating  the  likeli¬ 
hood  of  achieving  this  criterion  (the  p  ^5  level  in  this  case).  Hence,  equiva¬ 
lent  performance  in  test  is  indicated  by  a  probability  of  ,950  (one  equal  to 
the  reference),  superior  performance  by  a  probability  greater  than  .950  and 
inferior  performance  by  a  probability  less  than  .950.  Also,  as  before,  the 
probabilities  for  separate  parameters  were  taken  cumulatively  to  obtain  p  and 
taken  jointly  to  obtain  p  values  for  each  of  the  several  phases.  These,  in 
turn,  also  were  combined  as  before  to  obtain  still  more  inclusive  values  for 
individuals  performing  the  whole  mission  and  for  crews.  However,  since  the 
estimated  probability  of  achieving  the  criterion  in  this  case  is  .950  the 
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expected  value  for  a  given  £  is  something  less  than  .950,  as  a  direct  function 
of  the  number  of  separate  prolabilities  ufaich  are  thereby  taken  jointly.  For 
example,  the  expected  £  for  a  phase  represented  by  three  measures  is  .857. 

|>  remains  unaffected  and  the  expected  value  for  it  ia  always  .950. 

The  primary  results  of  this  analysis  of  test  mission  data  may  be  seen  in 
table  X  which,  like  the  parallel  table  VIII,  summarises  the  p  and  $  values 
obtained  by  phase  and  the  more  inclusive  indicants  baaed  upon  these.  (The 
probabilities  for  each  individual  and  each  parameter  are  tabled  in  appendix  HI. ) 
Table  X  shows  that  the  likelihood  of  pilots  in  C~8  and  0*13  achieving  their 
end-cf-t raining  p#Q(-  level  in  the  test  mission  was  nowhere  near  as  great  as 
their  likelihood  or  achiereng  the  hypothetical  criteria.  This  result  is  to  be 
expected  in  view  of  their  great  likelihood  of  meeting  the  criteria  at  the  end 
of  training.  Bit,  in  contrast,  the  pilots  of  0-4  and  0-9  had  about  the  same 
likelihood  of  reaching  their  p  nc  levels  as  they  did  of  reaching  the  criteria— 
again  to  be  expected  in  view  or their  lessor  skill  at  the  end  of  training. 

Once  again,  as  was  noted  in  the  prior  analysis,  the  individual  pilots 
showed  considerable  variability  in  their  capability  to  perform  in  test  at 
former  levels  on  the  several  phases.  This  is  illuatzated  by  the  two  examples 
ia  figure  9 — again  depicting  the  performances  of  P-8l  and  P-131,  but  now  with 
reference  to  the  p  qc  level  established  by  each  individually  rather  than  with 
reference  to  a  common  arbitrary  criterion.  Thus,  both  these  pilots  showed 
considerable  reduction  in  performance  of  certain  phases  and  not  of  others. 
However,  they  were  not  entirely  consistent  ia  this.  Although  both  had  diffi¬ 
culty  with  braking  and  hover,  docking,  and  earth  entry,  P~8l  also  had  difficulty 
with  the  translunar  insertion  and  the  separation  and  deorbit,  whereas  P-131  had 
difficulty  with  transposition.  This  pattern  of  variability  in  phase-to-phase 
performance  and  of  only  moderate  consistency  among  pilots  in  phase  performance 
is  typical.  Even  the  pilots  of  0-8  and  0-13  who  were  tested  second  and  third 
show  phase-by-phase  variations  in  test  performance  relative  to  former  levels, 
when  they  did  not  relative  to  criteria.  This  may  be  considered  confirmatory 
evidence  of  the  greater  sensitivity  to  performance  changes  that  analysis  by 
reference  to  former  individual  capability  brings. 

The  overall  mission  performance  probabilities  shown  in  table  X  also 
clearly  indicate  a  loss  on  the  part  of  all  individuals  and  crews  in  the  test 
mission,  over  capability  at  the  end  of  training.  Furthermore,  the  amount  of 
loss  seems  again  to  be  related  to  the  order  of  test — the  first  tested  individual 
always  having  performed  less  adequately  relative  to  his  previous  skill  than  the 
others.  These  relationships  oan  be  noted  more  conveniently  from  table  XI  in 
whioh  is  listed  for  each  pilot  the  amount  of  loss  in  probability  from  the 
expected  value  based  an  p  training  levels  and  the  percentage  of  the  expected 
value  such  a  loss  represents.  (All  values  in  this  table  indicate  losses  and 
so  are  not  signed.)  Thus,  for  example,  among  the  first  tested  piLota  deteriora¬ 
tion  in  test  from  individual  p  ^  level  expectancies  ranged  from  .228  to  .165 
(or  24  to  17.4$)  in  p.  Their  losses  in  p,  of  course,  are  even  greater — from 
46.3  to  22.3$.  All  showed  a  probability  of  <.0005  of  achieving  their  expected 
ji  value  and,  hence,  a  nominal  10Q$  loss  in  it.  However,  as  already  noted,  the 
losses  of  the  second  and  third  tested  pilots  are  never  so  great  (except  with 
respect  to  f>  which,  being  so  low  in  any  case,  does  not  discriminate  among  them). 
The  natural  result  is  that  the  mean  losses  of  the  crews  are  somewhat  less  than 
those  of  the  first  tostod  pilots — ranging  from  17.8  to  11.6$  in  p  and  39*0  to 
23.9$  in  5.  Hegarding  these  overall  losses  (both  of  the  first  tested  pilots 
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TAELS  X 


Probability  Estimated  from  Test  Mission  Performance  of 

Attaining  the  p  „  Skill  Level  Aohieved.  in  Training 
*95 

Crew/  Phase  Mission  v 


Pilot  TLl 

TEN 

LOI 

SIX)  EH 

LA 

Rend 

Xtok 

EE 

t.f  i 

0-4  1  (?)  — r 

.587 

.988 

.591  .672 

.991 

ND 

.331 

.994 

.769 

(?)  —  <.001 

.977  <.001  .001 

.991 

ND 

<.001 

.981 

.494  <.001 

2  .999 

.967 

.992 

.640  .463 

.993 

ND 

.994 

.862 

cS64 

.998 

.903 

.985  <.001  <.001 

.993 

ND 

.$81 

.630 

.686  <.001 

3  .995 

.975 

— . 

.803  .621 

— 

ND 

,966 

.731 

.886 

.990 

.926 

.999 

.504  .113 

- - 

ND 

,900 

.384 

.727  .018 

all  {?)*  .998 

.843 

.993 

.678  .585 

.995 

ND 

.764 

.862 

.840 

(?)  .99? 

.610 

.987 

.168  .036 

.995 

ND 

.627 

.665 

.636  .006 

C-9  1  (?)  .998 

.330 

.997 

ND  .250 

.998 

ND 

.967 

,956 

.785* 

(?)  .997  ■ 

<.001 

.994 

ND  <.001 

.998 

ND 

.903 

.870 

.680*<.001* 

2  .988 

.769 

.862 

.982*  .250 

ND 

ND 

.822 

.977 

.807* 

.977 

.314 

.725 

.982*<.00I 

ND 

ND 

.494 

.932 

,632*<,0C1* 

3  .497 

.989 

.998 

•931*  ,494* 

.997 

ND 

.963 

—  * 

.859* 

<.001 

.671 

.996 

.931*<»001* 

.997 

ND 

.892 

.686*<.001* 

all  (p)  .828 

.696 

•  952 

.956*  .331 

.998* 

ND 

.917 

.978* 

,817* 

(?)  .658 

.328 

.905 

,956*<.00l 

.9 98* 

ND 

.763 

.934* 

,666*<.001* 

0~3  1  (p)  .614 

.997 

.992 

.534  .723 

.933 

.969 

,394 

.442 

.733 

(?)  .231 

.991 

.984 

.134  .244 

,933 

.969 

.010 

.048 

.505  <.001 

2  .452 

.963 

.82? 

.933  .431 

.995 

_ 

.979 

.659 

.604 

.001 

.891 

.657 

.802  .  023 

.995 

~ 

.938  <.001 

,59c  <.001 

3  .994 

.965 

.888 

.553  .698 

.998 

.99? 

.653 

.520 

.807 

.989 

.896 

.786 

<.001  .005 

,998 

.997 

<.001 

.00? 

.520  <.001 

all  (p)  .687 

.975 

.902 

.673  .617 

.975 

.969 

.675 

,540 

.781 

(?)  .407 

.926 

.809 

.312  .091 

.975 

.989 

.316 

.018 

.533  <.001 

0-13  1  (p)  ~ 

.591 

.934 

.853  .285 

.382 

.991 

.653 

.755 

.122 

(?)  — 

<.001 

.968 

.600  <.001 

.382 

.991 

.026 

.301 

.474  <.001 

2  .712 

.939 

.966 

.979  .946 

.957 

— 

.996 

.644 

.871 

.424 

.819 

.931 

.937  .797 

.6.57 

— 

.989 

.025 

.731  .004 

3  .634 

.964 

.768 

.764  .558 

.981 

.983 

.885 

.710 

.805 

.278 

.893 

.572 

.362  .046 

.981 

.983 

.689 

.322 

.570  .001 

all  (?)  .782 

.831 

.906 

.865  .596 

.673 

.991 

,845 

.703 

.799 

(?)  .567 

.571 

.824 

,633  .281 

.673 

.9^1 

.568 

.216 

.592  .002 

t  dash  indicates  value  >.9995 

*  based  on  reduced  data 

ND  data  ware  unavai labia 


v  probabilities  for  "all"  pilots  are 
individual  values  taken  cumulatively 
(i»e.r  means  of  individual  values) 
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Loss  in  fast  Mission  2©lisMlity 

ffres  p  „  2m'iMEg  £e?al  « 

•X? 


Or®^/Meas»© 

A 

Pilot 

2 

3 

dll 

(assn) 

P 

(sspsefeea) 

4  p 

.181 

(19.1) 

.086 

(9.1) 

,064 

(6.7) 

.no 

(n.6) 

.950 

5 

.381 

(43.5) 

.189 

(21.6) 

.148 

(16.9) 

.239% 

(27.3) 

.875 

1 

.340 

(100.0) 

.340 

(100.0) 

.322 

(94.7) 

,•334^ 

(99.2) 

.340 

9  f 

.165 

(17.4) 

.143 

(15.1) 

.091 

(9.6) 

.133 

(14.0) 

.950 

$ 

.•195 

(22.3) 

.243 

(27.8) 

.189 

(21,6) 

.209 

(23.9) 

.875 

2 

P 

.340 

(100.0) 

(ICQ.O) 

,  .340 
(100.0) 

.340 

(100.0) 

.340 

8  F 

.217 

(22.8) 

.146 

(15.4) 

(i5ci) 

.169 

(17.8) 

.95© 

I 

.378 

(42.8) 

.•293 

(33.?) 

ci% 

.345x 

(39.0) 

.885 

t 

p 

.323 

(100.0) 

.323% 

(100.0) 

,  .323 
(1G0.Q) 

.323 

(100.0) 

.323 

13  p 

.228 

(24.0) 

•°79 

(8.3) 

.145 

(15.3) 

.151 

(15.9) 

.950 

I 

.409 

(46.3) 

,152 

(17.2) 

.313n 

(35.4) 

.291 

(33.0) 

.883 

P 

.323% 

(100.0) 

•3W 

(98.8) 

.322 

(99.7) 

.321 

(99.5) 

.323 

*  percentage  loan  indicated  in  pare&thasos 
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and  the  average  losses  of  the  orewa)  it  is  of  special  interest  that  the 
differences  among  crews  and  individuals  is  quite  small*  This  is  in  contrast 
with  the  previously  described  results  with  r-ofe^-enoe  to  criteria  in  whidh.  both 
0*4  and  0-9  showed  average  gains  and  in  which  one  first  tested  individual 
actually  showed  a  gain.  Thus,  since  apparently  considerable  difference  in 
skill  level  existed,  it  may  be  said  that  whatever  the  level  of  skill  acquired 
the  deterioration  in  reliability  from  that  level  or  relative  to  it  was  nearly 
constant. 

On  the  other  hand,  that  little  difference  among  individual  and/or  crew 
losses  was  found  argues  again  that  the  effects  of  retention  interval,  if  any, 
were  slight.  The  first  tested  pilots*  losses  in  reliabilities  are  ordered 
91<41<8l<131  in  both  3  and  $  while  those  of  the  crews  are  ordered  4<9<13<8  in 
f>  and  9<4<13<8  in  J.  Apparently,  although  the  differences  are  sisall  and  some*- 
what  inconsistent,  loss  in  reliability  did  tend  to  be  directly  related  to 
retention  interval. 

In  general,  the  results  of  this  additional  analysis  of  test  performance 
which  makes  reference  to  individual  capability,  parallel  those  of  the  first, 
which  makes  reference  to  hypothetical  system  criteria.  The  principal  differences 
in  outcome  which  can  be  noted  are  (l)  that  the  losses  in  capability  relative 
to  end~of*-training  levels  are  about  two  to  three  times  greater  end  (2) 
that  the  findings  for  0-4  and  C~9  are  not  in  complete  agreement,  as  would  be 
expected  on  the  basis  of  their  less  adequate  training.  Otherwise,  both  analyses 
clearly  indicate  a  degree  of  degradation  in  test  mission  performance  compared 
with  eneb-of -training  performance,  varying  considerably  by  phase  for  individual 
pilots  in  a  not  completely  consistent  fashion,  but  which  overall  phases  show 
relatively  evident  effects  of  the  order  in  which  the  pilots  were  tested  and 
relatively  slight  effects  of  the  retention  interval  involved. 


Retraining  Performance 

Having  shown  that  the  pilots  did  experience  a  loss  in  skill  over  the 
retention  period  the  next  question  is,  How  well  did  they  regain  their  skill  in 
the  subsequent  retraining  trials?  The  rapidity  with  which  they  achieved  their 
former  levels,  if  they  did  achieve  them,  would  carry  valuable  implications  as 
to  the  possibilities  of  mitigating  the  degrading  effects  of  lengthy  retention 
periods  in  operational  circumstances  which  warrant  the  concern.  Furthermore, 
because  even  the  pilots  of  0-8  and  0-13  had  not  necessarily  readied  the  limits 
of  their  skill  potential  in  training  (those  of  0-4  and  0-9  definitely  had  not) 
it  is  of  special  interest  to  ascertain  if,  possibly,  in  the  course  of  the 
retraining  given  them  they  would  not  only  regain,  but  surpass,  their  former 
levels.  In  fact,  it  was  just  this  possibility  that  prompted  the  3  days  of 
retraining  fer  0-8  and  0-13. 

With  these  two  questions  in  view,  the  analysis  of  retraining  performance 
paralleled  that  of  test  mission  performance,  focusing  attention  first  on 
estimated  likelihood  of  reaching  the  hypothetical  system  criteria  and  next  on 
estimated  likelihood  of  achieving  the  level  of  performance  achievable  95$  of 
the  time  at  the  end  of  original  training.  Because  (with  the  except  ion  noted) 
pilots  of  0-8  and  0-13  had  as  many  as  28  and  never  fewer  than  8  retraining 
trials  or.  certain  phases  it  was  possible  to  consider  their  data  in  more  than 
one  way.  Thus,  the  analysis  just  mentioned  was  first  performed  upon  their 
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initial  four-trial  block  of  retraining  data  and  then  repeated  upon  the  best 
four-trial  block  for  each  individual  and  phase  (f>  was  the  criterion  used  in 
selecting  the  best  trial  block)#  la  contrast,  because  the  pilots  of  0*4  and 
0-9  received  much  lees  retraining  (from  8  to  2  trials,  depending  on  phase)  aad 
the  data  for  these  trials  are  not  quits  complete  all.  their  available  retraining 
data  were  considered  in  a  single  analysis.  This  analysis  is  considered  roughly 
comparable  to  the  analysis  of  initial  four-trial  performance  of  0-8  and  0*13 
pilots. 

Eeliability_pe?_  Hgpothetjgal. Vitoria.  Accordingly,  the  results  of  the 
analysis  of  initial  retraining  performance  of  all  pilots  relative  tc  criteria 
are  summarised  together  in  table  XII.  As  in  previous  tables  of  this  nature  the 
|S  and  p  values  for  separate  phases  are  indicated,  followed  V  the  more  inclusive 
probabilities  referring  to  all  phases  and  the  whole  crew.  The  phase  reli¬ 
abilities  are,  of  course,  direotly  comparable  to  these  reported  in  table  VII 
for  original  training  and  in  table  VIII  for  the  test  mission.  However,  the  all- 
phase  probabilities  for  individuals  and  crews  are  not  directly  comparable  to 
the  mission  values  of  the  former  tables,  because  they  are  based  on  a  more 
limited  set  of  phases  (namely,  those  which  had  been  selected  for  retraining). 

Judging  from  the  phase  probabilities  reported  in  table  XII,  aa  before, 
the  pilots  showed  varying  capabilities  to  perform  the  several  phases.  They 
also  varied  considerably  among  each  other  in  capability  to  perform  any  particu¬ 
lar  phase— especially  the  pilots  of  C-4  and  0*9  •  However,  together  they  showed 
a  rather  consistent  tendency  to  perform  certain  phases  better  than  others. 

This  is  olearly  evident  in  the  values  for  0-8  aad  0-13  pilots — none  of  whom 
apparently  had  difficulty  with  the  transposition,  lunar  orbit  insertion,  aad 
docking  phases*  But  as  a  group  they  apparently  had  some  'difficulty  with  the 
separation  and  deorbit,  braking  and  hover,  and  earth  entry  phases.  This 
pattern  is  reflected  also  in  the  lower  crew  probabilities  for  0*4  and  0-9  for 
the  same  phases. 

In  general,  the  phase  probabilities,  both  for  individuals  and  for  crews, 
are  quite  high,  approximating  those  of  training  (table  VII )  and  surpassing 
those  of  the  test  mission  (table  VIIl),  as  would  be  expected*  Ones  again  the 
performance  levels  of  the  0-8  and  0-13  pilots  was  distinctively  superior  to 
that  of  the  0-4  and  0-9  pilots.  Furthermore,  when  the  best  four-trial  perform¬ 
ance  of  0-8  and  0-13  pilots,  shown  in  table  XIII,  is  considered  it  is  seen  that 
additional  opportunities  for  training  resulted  in  even  greater  performance 
reliability.  In  sore  than  two-thirds  of  the  phases  (23  out  of  33)  these  pilots 
mot.  or  exceeded  their  ondrof-training  levels.  A  graphic  illustration  of 
retraining  performance  may  be  seen  by  referring  to  figure  8  which  includes  a 
plot  of  p  values  obtained  for  P-8l  and  P-131,  along  with  the  comparable  plots 
representing  training  and  test  mission  performance. 

Going  beyond  the  pilots'  phase-byphase  performance,  a  more  complete 
view  of  their  retraining  achievements  may  be  had  by  examining  tables  XIV 
(for  0-8  and  0-13)  and  XV  (for  0-4  and  0-9),  In  the  left-hand  columns  in 
each  of  these  tables  are  summarised  the  f,  f  and  $  values  with  respect  to 
oriteria  for  individuals,  and  the  related  cumulative  probabilities  for  crews, 
in  training,  in  the  test  mission,  and  in  retraining.  The  probabilities  given 
for  training  and  test  mission  in  these  tables  wore  calculated  on  the  basis  of 
only  those  phases  on  which  retraining  ms  given  and  are,  therefore,  direotly 
comparable  to  the  probabilities  for  retraining.  (Having  been  computed  on  the 
basis  of  only  certain  mission  phases,  they  axe  not  the  same  values  as  are  given 
in  tables  VII  aad  vm.) 
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BE 

(8) 
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w 
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0-4  1  (?) 

*818 

*990 

ID 
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,368 

.707 

(?) 

.455 

.979 

ND 

.033 

.683 
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2 
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HD 

.723 
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.66? 
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.404 

ND 

.241 

.213 
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,211  <,001 

3 

.938 
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HD 

.907 
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.985 

.959 

.817 
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HD 

.627 

.934 

.956 
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.808 

.896 

ND 

.691 

.823 
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.778 
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.472 

.791 
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.300 
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-.337 

.502  .151 
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.724 

.766 

HD 

.485 
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.550 

,651 

(?) 

.326 

.587 

HD 

.048 

.353 

.118 
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<*001 

2 

.817 

.665 

HD 

.616 

.881 

.863 

.768 

.452 

.414 

HD 

.100 

.680 

.631 

.455 

.008 

3 

.765 
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HD 

.554 
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HD 
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.775 

.815 

.951 

.967 

.950 

.782 

.964 

.481 

.630 

.837 

.807 

.729 

.149 

.78$ 

.002 

.004 

.288 

.973 

.947 

.972 

.995 

.791 

.850 

.937 

.928 

.902 

.922 

.984 

.516 

.635 

.808 

•616 

.467 

.682 

.904 

.005 

.02 0 

•292 

**  Meed  on  th©  six  phases  of  retraining  (i.o.  >  512? (  LOI*  8$0*  13*  XCK*  J3) 
Hbtes  expected  values  for  5,  sard  $  por  p#95  level  are  »95®f  .858, 
and  .397 

#  hosed  on  incomplete  data 

t  parenthetical  values  obtained  ly  eusulsting  erevr  values  for  phases* 
instead  of  pilot  values  for  mission.  fhegr  differ  booonec  data  fur 
P-8l  ©ro  inooaplote 
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Tmss  xv 


Comparative  MLi&Mlif.ies  of  Grew©  Ch4  and  0*9  in 


Pilot 


0»4  X 


all 


0-9  1 


all 


“Training,  Seat  and  Hatraiaisg  *** 

Per  Criteria  in* 

Per  s  t 
I©V®i 

PS* 

Trag 
(lost  4) 

Sissicn 
Sest  ■ 

Eetsjg 

Eiegien 

Test 

Batmg 

.614 

.605 

.707 

.714 

.878 

.256 

.332 

.430 

.392 

.646 

<.001 

<.001 

<«O01 

<,001 

.084 

.674 

.763 

.66? 

.856 

.782 

.440 

.532 

.211 

.700 

.546 

<.001 

.003 

<.C0X 

<.001 

.001 

.788 

.742 

•959 

.859 

.888 

.48? 

.408 

.865 

*664 

.658 

.001 

<.001 

.453 

.036 

.025 

.602 

.703 

.778 

.810 

<849 

.395 

.424 

.502 

.585 

.617 

<.001 

.001 

.151 

.012 

.037 

.611 

.564 

.651 

.700 

,87 6 

.198 

.293 

,286 

.553 

,680 

<.001 

<.001 

<.001 

<.001 

.046 

•898 

.647 

.768 

.736 

.659 

.339 

.309 

.455 

•493 

.649 

.001 

<„00l 

.008 

<.001 

*035 

.665 

.731* 

.689* 

.889 

.861* 

.422 

.453* 

.367* 

.712 

.731* 

<.001 

.003* 

.001* 

<.001 

,027* 

.658 

.647 

.703 

.775 

.865 

.320 

.352 

.369 

.586 

.687 

<.001 

.001 

.003 

<.001 

.036 

**  based  oa  the  five  phases  of  retraining  (i.e.,  TMt  M,  SOX,  SSI,  M) 
uith  LOI  substituted  for  TEL  in  th©  tost  mission,  Izpseted  values  for 
f,  end  $  par  leval  ore  .950?  .858,  and  .483,  respoetively 

*  based  on  incomplete  data 
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Prom  table  XXV  it  is  evident  that  so  far  as  the  hypothetical  system 
criteria  are  concerned}  the  pilots  of  0-8  said.  0-13  quickly  regained  their  over¬ 
all  effectiveness  during  the  retraining  provided.  Pour  of  the  sis  had  already 
exceeded  their  ond-of-t raining  levels  in  their  first  four  retraining  trials  and 
all  virtually  met  or  exceeded  them  in  their  best  four-trial  block  of  retraining. 
The  cumulative  probabilities  for  both  crews  naturally  reflect  this  also  by 
indicating  gains  in  the  best  four  trials  with  respect  to  all  aspects  (f,  p,  J>) 
of  overall  effectiveness.  Some  individuals — for  example,  P-82,  P-131  and  P-132 — 
showed  especially  noteworthy  improvement®  Also,  it  is  likely  that  had  not  P~8l 
been  somewhat  disturbed  by  the  requirement  to  terminate  his  participation  after 
just  one-half  day  of  retraining  he  would  have  performed  still  more  effectively, 
with  the  result  of  greater  demonstrated  gain  for  himself  and  his  crew.  Of 
course,  the  estimated  reliabilities  of  all  the  pilots  were  very  high  in  the 
best  four-trial  block  in  retraining—^  ranging  from  .99$  to  .9$1»  !>  from  ,999 
to  .964,  and  i>  from  ,999  to  .789.  That  these  reliabilities  reflect  aome  gain 
over  original  training  implies  that,  as  was  surmised,  these  pilots  had  not  quite 
reached  the  limit  of  their  skill  potential  at  the  end  of  training. 

The  overall  effectiveness  in  retraining  of  0-4  and  0~9,  as  shown  in 
table  XV,  runs  parallel  to  that  of  0-8  and  0-13  in  some  respects  and  in  others 
it  does  not.  The  pilots  of  0-4  and  0-9  (P**42  excepted)  also  regained  their 
former  levels  of  reliability  in  the  brief  retraining  afforded  them.  Some, 
notably  P-41,  P“43$  and  P-92,  even  showed  substantial  gains.  Similarly,  the 
cumulative  probabilities  for  the  two  crews  reflect  this  general  improvement  in 
all  aspeots  of  overall  effectiveness.  However,  the  estimated  reliabilities 
with  respect  to  criteria  of  0-4  and  0-9  pilots  at  their  beat  were  still  generally 
low  whan  compared  with  those  of  the  0-8  and  0-13  pilots,  p  ranged  from  .959 
to  .651,  5  from  .865  to  .286  and  £  from  #453  to  <.001.  Clearly,  moat  of  the 
0-4  and  0-9  pilots  had  not  even  approached  the  limits  of  their  skill  potential 
in  original  training  and  did  not  do  so  in  retraining  either. 

Capability  Helative  to  Previous  XjSvel.  However,  when  the  retraining 
performance  of  the  pilota/craws  is  examined  in  terms  of  the  likelihood  of 
achieving  their  md-of-i raining  p#oc  level  a  somewhat  different,  although 
generally  oomplansntary,  view  is  obtained.  The  phase-by-phase  probabilities 
of  achieving  this  former  level  are  summarized  for  the  initial  retraining  of  all 
pilots  in  table  XVI  and  for  the  best  four-trial  blocks  of  0-8  and  0-13  pilots 
in  table  XVII.  (Once  again,  although  the  p  values  to  be  expected  are  .950  the 
£  values  to  be  expected  vary  with  the  number  of  measured  parameters  for  the 
phase,  as  in  the  comparison  table  X  concerning  test  mission  performance.)  In 
addition,  the  directly  comparable  figures  for  overall  effectiveness  in  the  test 
mission  and  the  expected  values  may  be  seen  in  the  right-hand  columns  of  tables 
XIV  and  XV,  for  0-8  and  0-13  and  0-4  and  0-9,  respectively. 

The  phase  probabilities  for  initial  retraining  given  in  table  XVI  ar© 
often,  but  certainly  not  always,  higher  than  those  for  the  test  mission  given 
in  table  X.  fact,  only  on  the  braking  and  hover  and  docking  phases  are  the 
cumulative  probabilities  for  crews  consistently  higher.  PVrthermore,  the 
probabilities  based  on  initial  retraining  seldom  reach  the  expected  value 
U950>  etc.)  thus  indicating  that  the  individuals  and  crews  performed  few  phases 
initially  at  their  former  levels  of  skill.  None  of  them  attained  their  former 
level  in  separation  and  deorbit,  braking  and  hover,  and  docking# 
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Again,  there  was  considerable  variation  within  individuals  in  their  capa¬ 
bility  to  perform  the  several  phases  at  their  former  level  and  among  individ¬ 
uals  in  their  capability  to  perform  a  given  phase  at  former  levels.  Overall, 
in  contrast  with  the  analysis  in  terms  of  oriteria,  their  effectiveness  on  the 
several  phases  does  not  show  a  very  consistent  pattern  of  effectiveness  favor¬ 
ing  certain  phases  over  others,  although  pilots  of  0-8  are  still  seen,  to  have 
performed  less  competently  on  the  same  three  phases  identified  earlier.  In 
general,  th®  phase-by-phase  probabilities  of  achieving  the  former  level  do 
not  seem  to  vary  as  greatly  as  did  those  of  meeting  the  hypothetical  oriteria. 

Hhsn  contrasted  with  the  entries  of  table  XVT  those  of  table  XVII  show, 
as  in  the  oriterial  analysis,  the  effects  of  the  additional  retraining  received 
by  the  pilots  of  0-8  and  0-13.  As  crews  in  the  best  four  trial  block  of  retrain¬ 
ing  they  surpassed  their  initial  four-trial  levels  in  nine  out  of  twelve  instances. 
However,  even  these  most  skilled  pilots  at  their  bast  did  not  consistently  exceed 
their  training  level  and  in  several  phases  were  far  short  of  it.  This  is  illus¬ 
trated  in  figure  9  in  which  retraining  performances  of  P-8l  and  P-131  are  plotted 
with  reference  to  their  former  p  qc  levels.  The  result,  in  overall  effective¬ 
ness,  was  that  only  one  of  the  six  (F-132)  slightly  exceeded  the  expected  value 
and  one  other  approximately  matched  that  value  (see  table  XIII).  The  discrepancy 
between  the  values  for  P-81  and  those  of  the  other  pilots  again  may  be  most 
easily  attributed  to  the  adverse  circumstances  under  which  ho  performed.  From 
these  findings  it  may  be  argued  that  the  pilots  of  0-8  and  0-13  had  indeed 
reached  their  ekill  potential  at  the  end  of  training— a  view  which  seemingly 
contradicts  the  earlier  findings  on  performance  with  respect  to  criteria. 

Having  noted,  then,  that  0-8  and  0-13  did  not  much  surpass  their  former 
levels  it  naturally  might  be  supposed  that  the  less-skilled  0*4  and  0~9  ware 
even  further  from  their  former  levels  in  retraining.  Sit,  the  last  column  of 
table  XIV  shows  that  this  was  not  so,  that  if  anything,  C~4  and  0-9  surpassed 
0-8  and  0-13  in  capability  to  perform  at  former  levels  in  the  initial  retrain¬ 
ing.  This  observation  too  is  at  variance  with  the  findings  on  retraining  perform¬ 
ance  with  respect  to  criteria. 

In  general,  concerning  retraining  it  is  concluded  that  in  the  first  few 
trials  most  pilots  regained  considerable  skill— enough  to  demonstrably  surpass 
their  test  mission  capability — but  not  enough  to  match  their  end-of-t raining 
levels.  The  amount  by  which  they  fell  short  of  the  p  qc  training  level  (from 
.250  to  .061  in  p,  excluding  P-8l)  was  surprisingly  similar,  irrespective  of 
their  degree  of  original  ekill.  Those  receiving  additional  retraining  approached 
and,  in  at  least  a  third  of  the  cases,  matched  their  former  levels  but  showad 
little  tendency  to  surpass  them.  End-of-t raining  capability  to  meet  hypothetical 
system  criteria  was  fully  regained  by  virtually  all  the  pilots  in  a  few  retrain¬ 
ing  trials,  several  pilots  actually  surpassing  it,  end  was  always  surpassed  in 
the  best  retraining  performance  of  those  receiving  additional  retraining.  But, 
the  less  wall-trained  pilots  of  0-4  and  0~9  continued  in  retraining  to  be 
distinctively  3e«s  able  to  meet  the  criteria  than  the  pilots  of  0-8  and  C~13, 
even  though  they  performed  equally  well  relative  to  their  own  former  levels. 

Evidently  then,  the  first  question  as  to  how  quickly  skill  was  reacquired 
may  beat  be  answered  in  this  ways  both  skilled  and  relatively  unskilled  pilots 
were  able  to  regain  their  capability  to  meet  the  hypothetical  system  criteria 
within  about  four  trials  following  the  test  mission,  but  not  their  full  original 
capability.  The  originally  well-trained  pilots  did  approximate  or  equal  their 
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TABLE  XVII 


Probability  Estimated  from  Best  Fcur-Trial  Block  In  Eetraining 
of  0*8  and  0-13  Attaining  the  p  Skill  Laval  of  the  Last  Pour 
Training  Trials 


Phase 

Crew/  All  Phases  v 

Pilot  TIT  LOI  SDO  BH  Bok  BB  gtg  £ 


0-8  1  i 

(?) 

ITD 

ND 

.885 

.664 

ND 

.383 

.644* 

i 

($) 

m 

ND 

.668 

.107 

IIP 

.051 

.275* 

.004* 

2 

•999 

.832 

.800 

.715 

.854 

.978 

.863 

.997 

.663 

.401 

.173 

.574 

.933 

.624 

.025 

3 

.482 

.950 

.920 

.623 

.829 

— 

.048 

.804 

.771 

.161 

.631 

.005 

all 

(p)v 

.916* 

.722 

.77  6 

.887* 

.661 

.779 

! 

.998* 

.832* 

.372 

.361 

.672* 

.382 

.510* 

.011* 

0-13  1 

(?) 

.999 

.886 

- 

.883 

.788 

.796 

.892 

(?) 

.997 

.771 

— 

.534 

.366 

.451 

.686 

.068 

2 

.962 

.986 

— 

.936 

— 

.932 

.969 

.887 

.972 

— — 

.757 

.799 

.902 

.521 

3 

.999 

.999 

.940 

.874 

— 

.893 

.951 

.996 

.999 

.821 

.500 

.705 

.837 

.288 

all 

(?) 

.987 

.957 

.980 

.898 

.929 

.874 

.937 

(?) 

.960 

.914 

.940 

.597 

.789 

.652 

.80 8 

.292 

t  dash  indicates  value  >. 9995 
*  based  on  reduced  data 

v  probabilities  for  "all”  pilots  are  individual  valtioe  taken 
cusulatively  (i.©.?  msm  of  individual  values) 
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full  original  capability  with  varying  smarts  of  additional  rotmining  3a  tfca 
savors!  phases*  la  contrast,  the  second  question  as  to  whether  the  pilots  of 
0-8  sad  0-13  wars  fully  trained  appears  unanswerable  on  the  basis  of  the  bare 
results  so  far  presented*  For,  on  the  one  Mad,  if  capability  to  East  sayst m 
criteria  is  tho  deciding  factor  they  east  be  considered  not  to  hava  b&m  folly 
trained  originally.  Os  the  other  hand,  if  capability  to  perform  at  a  highly 
reliable  level  (p.^)  dstesained  frcs  their  own  perfojsseaee  is  the  deciding 
factor  there  is  nohasis  for  supposing  that  they  were  not  fully  trained. 

In  the  following  section  of  the  report  this  problemtical  aspect  of  the 
findings  along  with  certain  other  aspects  of  the  study  which  bear  on  tbs 
interpretation  of  the  data  will  be  discussed  end  so sa  definitive  conclusions 
will  be  stated. 
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The  problem  of  interpreting  the  data  obtained  and  ths  analytic  results 
presented  actually  involves  consideration  of  at  least  ihr^a  sosediat  different 
kinds  of  factors,  These  ere,  first  of  all*  the  obvious  procedural  factors 
relating  to  the  Renner  cf  obtaining  the  data.  Lack  of  standardisation  in  pro- 
oedore  or  failure  to  control  or  randomise  soma  significant  variable  affecting 
ths  performance  of  the  test  personnel  can  certainly  bias  results*  In  order  to 
interpret  ths  study,  then,  it  is  desirable  to  consider  tha  possibilities  for 
bias  in  this  regard  and  to  gauge  their  likely  impact  on  the  data* 

Beyond  the  details  of  data  collection  another  aspect  of  the  study  signif¬ 
icantly  affecting  the  interpretation  of  findings  concerns  the  details  of  the 
analytic  procedures  used*  It  is  possible  that  sons  characteristics)  of  the 
analytic  method  employed  cay  operate  to  introduce  a  bias  in  the  representation 
or  comparison  of  performance  obsorvedr-thue  resulting  in  an  artifact,  or 
spurious  effect.  Any  possibilities  of  this  nature  should  also  be  considered 
and  their  implications  for  the  results  determined* 

Finally,  beyond  those  procedural  details  of  data,  collection  and  analysis 
there  are  always  the  a  rasa-ghat  broader  questions  of  gsneralisability— of  axtvuat 
to  which  the  findings  are  applicable  to  other  situations*  Although  generalises- 
bility— depends  greatly  on  the  specific  procedures  used,  it  also  depends  upon 
more  general  matters  relating  to  the  very  conceptualization  of  the  study* 

These,  too,  should  be  examined,  in  retrospect,  in  order  to  arrive  at  a  fully 
oonsidered  interpretation  of  the  results* 

Such  of  these  three  kinds  of  factors  will  be  discussed,  in  turn,  in  ths 
following  portions  of  this  sect  ion.  Finally,  in  keeping  with  those  interpreter 
tive  remarks  and  tho  analytic  results,  speoifio  study  findings  will  be  stated. 
Headers  who  do  not  care  to  follow  the  detailed  examination  of  the  nuasrous 
interpretative  factors  involved  say  turn  dirsotly  to  page  91  for  tho  statement 
of  findings*  Alternatively,  just  tho  more  specific  considerations  of  data 
collection  and  analysis  procedures  assy  be  skipped  by  turning  to  page  84  on  which 
the  discussion  of  general  inability  begins. 


Missing  Rita*  With  a  study  of  tho  magnitude  of  this  on©  in  sheer  voluao 
of  data  it  is  a  ocsson  occurrence  that  ocas  of  the  data  are  lost— usually  for  a 
variety  of  reasons.  Ecnce,  it  is  a  fair  question  to  ask  regarding  this  ono, 

To  what  extent  wore  data  lost  and  how  do  these  losses  affect  tho  results?  This 
question  eon  bo  answered  in  a  very  general  way  but  of  ooro  use  is  spsolfio 
information  on  shore  tho  losses  wore  ok  countered* 

Starting  then  with  tho  training  data,  and  restricting  concern  as  always 
to  flight  control,  tho  records  of  training  porforesne^  sahibi-Sod  by  CSredsky 
©t  al  (1966b)  soon  quite  oosplet®  for  tho  purpose  of  this  analysis.  Etas,  m 
also  say  bo  noted  in  table  XS7  of  appendix  HI  (present  report),  caffioieffltt  data 


oa  0-8  said  0-13  were  available  to  dsnya  provide  a  good  foul-trial  taels  for 
©stieating  cadf-of- training  performance.  la  fact,  with  these  two  carers  la  only 
two  cases  (P-83  on  TLX  and  P-131  cm  LOl)  was  these  an  evident  difficulty  requir¬ 
ing  attention  and  this  involved  possibly  erroneous?  not  Kissing  data*  In  these 
two  instances  the  last  four  values  included  one  will  oh  was  obviously  extreae,  as 
cohered  with  the  whole  training  record  of  the  individual*  Consequently?  to 
obtain  a  fair  estimate  of  capability  these  two  values  wer®  replaced  by  reaching 
one  trial  further  beck  in  th®  training  record*  There  were  a  few  other  instances 
where  values  which  seamed  unrepresentative  of  the  individaal's  cad-of-trsiniag 
capability  tr®r®  noted  but*  sins®  they  were  not  obviously  extreme  relative  to  the 
Thole  record  they  wore  used*  Perhaps  the  most  notable  each  instances  are  those 
of  P-132  on  8B0  (velocity  end  yaw)  and  oa  Xbk  (displacsssat  rate)  and  P-133  °a 
M  (velooity).  In  these  instances  it  is  Quite  possible  that  the  estimate  of 
training  is  an  underestimate  ssid  the  comparisons  of  training  with  other  perform¬ 
ance  are  consequently  biased*  These  possibilities  will  be  explored  more  precisely 
in  the  discussion  of  sampling*  Meanwhile?  anticipating  that  discussion?  it  is 
doubtful  that  beyond  affecting  the  estimates  of  original  capability  of  P-132  and 
P-133  the  extrsE©  values  entered  greatly  in  th®  analysis  of  0-8  and  0»13  perform¬ 
ance. 


In  contrast?  the  record  for  0-4  and  0-9  is  not  nearly  so  complete  although? 
considering  their  evident  lack  of  training  and  th©  secondary  interest  to  be 
accorded  their  data?  it  still  seeas  quite  adequate*  It  war,  necessary  sessatimas 
to  reach  bade  more  than  four  trial*  in  training  to  secure  a  basis  for  estimation 
or  to  accept  loss  than  four  trials  as  a  basis?  but  this  did  not  occur  to  a  great 
extent*  As  table  XXV  indicates?  acceptable  estimates  were  possible  for  all  but 
the  r  idszvous  performance,  the  S30  performance  in  yaw  and  pitch  of  0-9,  ©ad  th© 
LPA  performance  of  P-92.  Bsadssvous  performance  cannot  be  ©stisatod  ®»r®ly  be¬ 
cause  fuel  osuapticn  for  it  was  not  at  that  time  being  distinguished  frost  that 
for  docking  in  th®  performance  records*  This  lade  of  information  on  rendezvous 
doe©  force  discounting  ren&ssvcus  in  th©  record  of  0-3  a  X  0-13  before  comparison 
of  the  two  sets  of  crotsx  (as  noted  previously),  "tent  it  otherwise  hm  little  bear¬ 
ing  on  the  result©.  The  other  losses  mentioned  also  sesa  of  sinoi-  otmo&m — merely 
a  nuisance  in  calculation  and  in  oro 3 sj-coaparison . 

Th©  test  mission  data  are?  in  a  sense?  even  cor©  complete  than  the  training 
data.  Only  the  rendezvous  informatics  for  0-4  and  0-9  (for  the  reason  already 
noted),  the  velooity  control  of  P~91  in  SK>  and  th®  displacement  of  P-93  in  F*S 
ar©  unavailable.  However?  the  lack  of  information  on  training  performance  o» 

0-9  in  yaw  and  pitch  in  SEO  and  of  P-92  in  LPA  as  a  basis  for  estimating  varia¬ 
bility?  additionally  saadn  calculation  of  test  mission  probabilities  for  these 
performances  impossible  as  well.  Again  though?  considering  the  secondary  interest 
in  0-4  and  0-9?  the  net  effect  of  these  data  losses  seems  slight. 

Similar  general  conclusions  concerning  the  retraining  data  appear  appro¬ 
priate.  For  0-8  and  0-13  the  only  losses  were  in  the  performance  of  F-8l.  These 
losses  were,  of  course?  considerable  and  they  restrict  opportunities  to  make 
fruitful  comparisons  involving  the  retraining  performance  of  this  ©specially 
significant  first-tested  pilot.  But  the  sheer  loos  in  data  is  probably  not  the 
main  concern  (as  will  be  brought  out  core  folly  in  a  following  discussion)?  and 
the  principal  interest  in  retraining  is  with  respect  to  given  individual's  gains 
or  losses  over  other  performances.  There  are?  of  course?  no  data  for  retraining 
performance  of  0-4  and  0-9  on  SIX)  and  LOI  (TEC  being  substituted)  which  makss 
for  nuisance  in  crcso-cosrparison?  but  those  are  difficulties  arising  by  design 
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rathar  than  by  leas,  as  such.  There  are  instances  (appendix  table  XX¥HX)  in 
which  the  estimates  for  pilots  of  CM-  and  0-9  had  to  to  based  ca  re&iesd  data 
owing  to  a  lack  of  information  on  a  few  parameters  and  trials  aad  on©  case 
(F~93  on  TEE,  V)  for  whi&h  no  estimate  waa  possible. 

Thus,  in  general,  although  the  instances  of  missing  data  noted  did  cause 
some  nuisance  in  analysis  and  do  make  cress-comparlgon  acre  complicated  than 
with  a  coapleta  record,  they  certainly  do  not  make  meaningful  interpretation 
impossible  or,  if  properly  toadied,  Mas  the  results#  Perhaps  tha  most  serious 
of  theas  considerations  is  the  problem  (to  be  discussed  in  sore  detail)  of 
possible  bias  due  to  the  inclusion  of  unrepresentative  vines— hut  this  problem 
is  certainly  not  unique  to  this  study* 

Conduct  of  the  Test.  Apart  from  the  broader  issues  of  simulation  involved, 
there  are  a  number  of  particulars  of  the  my  in  which  this  test  ms  conducted 
which  might  possibly  have  influenced  the  results*  In  the  first  place,  in  order 
to  make  efficient  me  cf  personnel  tia®  and  minimise  costs  involved,  the  full 
7-day  mission  was  compressed  into  a  single  work  day  by  eliminating  long  coast 
periods  and  navigational  tasks.  In  fact,  it  was  compressed  oven  more  than  that 
by  arranging  for  three  performances  of  each  min  phss3“”cach  of  the  three  crew- 
members  occupying  each  ormr  position  once.  What  was  th®  probable  effect  of  such 
sever©  compression  on  the  performances  of  th®  pilots  relative  to  what  they  might 
have  done?  This  question  is  naturally  difficult  to  answer  end  perhaps  frankly 
ought  to  to  resolved  on  th©  basis  of  comparison  data.  However,  lacking  such  data 
and  yet  toing  appreciative  of  the  possible  significance  of  the  compression  factor 
com  estimation  appears  desirable.  What  then,  in  tense  of  known  psychological 
variables,  would  to  the  likely  effects  of  time  compression? 

In  view  of  the  available  information  on  work  periods  and  th®  effects  of 
continuing  work  it  readily  may  to  supposed  that  as  the  test  day  wore  on  the 
pilot®  baesm©  somewhat  less  able  to  perform.  This  downward  trend  would  to 
expected  to  continue  until,  toward  the  very  last  portion,  the  anticipation  of 
completing  th®  predefined  task  resulted  in  an  end  spurt.  Similar  trends,  but 
of  shorter  duration,  might  also  be  expected  within  the  morning  until  lunch  was 
provided  to  the  pilots  in  the  simulated  crew  compartment  and,  possibly,  within 
a  given  mission  cycle  for  0-8  and  0-13 »  the  pilots  of  whoa  completed  a  whole 
mission  in  the  same  position.  As  the  net  result  of  such  effects  all  the  pilots 
(of  all  crews),  ao  individuals,  might  to  expected  to  exhibit  parallel  trends 
downward  within  th®  mission  sequence  until  the  last  phaa®,  with  a  slight  improve¬ 
ment  in  any  phase  performed  just  before  lunch.  (For  0-4  and  C-9  the  phase 
performed  before  lunch  was  LQ1  but  for  0-8  and  0-13  th®  first-tested  pilot 
completed  23  just  before  lunch. )  However,  overlaid  upon  these  predictable  min 
trends  are  a  number  of  virtually  indeterminate  possible  affects  on  individuals 
as  might  arise  from  happenstance.©  in  scheduling,  which  afforded  break  periods 
in  sporadio  fashion  and  having  to  wait  out  relatively  long  periods  of  inactivity 
within  the  mission  sequences  of  0-8  and  C-13.  (in  fact,  one  0-13  pilot  was 
observed  showing  all  the  typical  signs  of  great  boredom  while  waiting  his  turn 
in  aidafternoon. ) 

On  the  other  hand,  it  is  also  known  that  such  trends  as  those  are  largely 
absent  from  the  performance  of  highly  iaeSe-eotivated  individuals  as  it  is  be¬ 
lieved  these  pilots  wore.  These  work  dscreasnt  effects  are  also  diminished 
when  the  work  is  of  a  varied  rather  than  repetitive,  routine  nature.  Further- 
more,  the  overall  work-period  (about  13  hours)  was  not  so  long  as  to  bring  about 


sufficient  fatigue  in  these  reasonably  rested  pilots  to  coeacica  mob  drami, 
if  strong  task  motivation  is  assumed,  for  there  vers  May  opportunities  for  rest 
during  th®  workday. 

However,  of  perhaps  store  concern  than  the  possible  trends  with  continuing 
work  are  th®  possible  effects  of  task  interference  arising  frost  the  recjairsEsat 
to  perform  the  tasks  in  close  succession.  This  would  ba  of  special  concern  in 
the  performance  of  0-8  and  0-13,  the  pilots  of  which  quickly  passed  fro®  one 
phaa©  to  the  other.  Over  the  longer  periods  between  phase  performances  of  0-4 
and  0-9  the  effects  would  diminish,  (torrent  vaderstaadiag  of  transfer  phenomena 
of  such  a  nature  predicts  that  the  effect  will  ba  greater  to  the  extant  that 
responses  to  the  sa m  or  similar  stimuli  in  the  two  situations  of  concern  are 
antagonistic.  But,  sine©  the  exact  character  of  the  intertask  similarities  can 
seldom  be  specified  a  prioi,  precis®  prediction  of  the  effect  is  seldom  possible 
and,  therefor®,  again  lacking  comparative  data,  little  beyond  the  minimal  supposi- 
tion  that  there  may  have  boon  some  intertask  interference  esess  warranted.  It 
may  also  b®  supposed  on  the  grounds  of  greater  possibility  for  antagonistic 
response  that  this  effect  would  be  most  pronounced  in  the  first  phase  perfoxssod 
at  on©  statical  after  having  performed  at  another  (i„@„,  cn  transferring  stations). 
Of  course,  ffeoilitstivs  effects  might  also  occur  to  the  degree  of  positive  inter¬ 
task  similarity  in  both  stimuli  and  responses  and  they  would  be  expected  to 
parallel  those  of  real-time  operation  with  enhanced  magnitude.  On  the  other 
hand,  such  facultative  and  interference  effects  as  these  are  known  to  diminish 
with  increased  skill  and  with  the  frequency  with  which  the  tasks  involved  have 
been  successively  performed.  All  the  orews  had  been  required  to  shift  from 
practice  on  one  phase  to  another  rather  rapidly  in  the  latter  portion  of  train¬ 
ing  and,  of  course,  0-8  and  0-13  did  so  even  more  and  were  trail-trains d  by  usual 
standards.  Thus,  it  is  conjectured  that  although  sc m  additional  facilitation 
and  interference  owing  to  the  close  succession  in  which  the  mission  phases  were 
performed  may  have  occurred,  it  is  not  likely  they  wore  of  sufficient  magnitude 
to  bias  the  min  trend  of  the  results.  Th®  effects  of  this  nature,  if  existing 
at  all,  would  be  small  changes  up  or  down  in  parameter  and  phase  reliabilities. 

The  multiple  testing  of  the  orews  so  that  each  crawasabsr  performed  in 
sack  position  and  the  necessarily  unique  order  in  which  this  was  accomplished 
represents  another  significant  factor  bearing  ca  the  results.  However,  rather 
than  being  forced  to  depend  upon  speculation  about  this  factor  the  analytic 
method  used  permits  direct  assessment  of  most  of  the  effects  of  concern.  These 
have  already  been  presented  and  provide,  in  general,  a  clear  indication  that 
the  second-  and  third-tested  individuals  benefited  frost  their  imasdiate  p re- 
exposure  to  the  mission.  As  already  noted,  this  factor  might  have  interacted 
with  the  work  decrement  factor  to  produce  some  differential  effects  on  0-8  and 
0-13  performance,  but  if  the  work  decrement  was  slight  these  differential  effects 
would  also  ho  slight. 

The  additional  differences  in  order  saong  the  two  sots  of  crews  (0-4  and 
0-9  va  0-8  and  0-13)  deriving  from  the  different  way  in  which  multiple  testing 
was  handled  within  the  mission  also  might  have  produced  some  effects.  Because 
of  the  groat  difference  in  performance  level  of  the  arows  in  the  two  study  phases 
no  deductions  from  the  data  concerning  such  offsets  scorn  possible  However,  it 
my  be  surmised  that  if  such  effects  occurred  they  ware  relatively  slight  end 
variable,  and  dependent  upon  the  concerted  influence  of  the  continuing  work  and 
task  transfer  factors  already  discussed. 
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Another  aspect  of  the  testing  procedure  deserving  special  concern  is  the 
diett2rtsn.ee  in  the  test  schedule  of  0-8*  necessitated  by  3Mfr*s  unplanned 
imminent  departure  on  special  asei^imat.  Available  inf  oration  on  this  require" 
meat  was  transmitted  to  P-8l  in  the  initial  portion  of  Ms  (0-8*8  initial) 
mission.  On  completing  the  mission  (near  noon)*  and  feeing  uncertain  as  to 
exactly  when  ho  would  have  to  leave*  P-8l  left  the  simulated  crew  station* 
donned  street  clothes*  ate  lunoh  in  haste,  and  thus  made  all  departure  prepare." 
tlons.  Consequently,  he  was  not  available  the  balance  of  the  day  to  perform, 
in  turn*  as  engineer  and  navigator  in  the  subsequent  test  missions. 

The  consequences  for  the  performance  record  of  0»8  are,  of  course,  diffi¬ 
cult  to  gauge.  But,  it  may  at  least  fee  guessed  that  the  probable  effects*  if 
any,  were  to  degrade  the  performances  of  all  the  crewmembers.  P-81,  informed 
in  midaissicn  of  a  requirement  of  considerable  personal  importance*  may  well 
not  have  attended  as  fully  as  he  might  have  to  mission  tasks.  P-82  and  P-83 
were  required  to  perform  under  an  arrangement  neither  standard  nor  expected 
(though  not  unusual  in  their  training  experience)  and  may  wall  have  wperienoed 
minor  confusion  with  perhaps  some  loss  of  motivation  through  disruption  of  crew 
integrity.  However,  if  the  performance  of  0-8  in  the  test  mission  was  degraded 
in  this  manner  it  is  euspsotod  that  the  degradation  was  slight,  primarily  again 
because  of  the  apparently  still  excellent  task  motivation  and  professional  in¬ 
terest  of  the  pilots.  The  disturbance  in  procedure  represented  mostly  a  nuisance 
to  be  accommodated  for*  not  unusual  in  their  experience. 

As  it  turned  out  P-8l  did  not  depart  the  first  test  day  hit  was  informed 
(upon  being  called  off  a  commercial  aircraft  about  to  leave  the  passenger  ter¬ 
minal  for  takeoff)  that  he  should  stay  one  day  longer,  This  ho  did  and  was 
thus  able  to  engage  in  retraining  for  one-half  day  before  actually  departing. 
Unfortunately,  in  having  encountered  still  another  change  in  plans,  with  what 
he  interpreted  as  a  possibly  largo  personal  sacrifice  in  his  career  and  having 
not  been  able  to  recover  his  luggage,  his  motivation  for  the  retraining  suffer© dr- 
to  say  the  least.  Exactly  how  suoh  better  he  would  have  performed  under  the 
originally  planned  circumstances  is  uncertain  but*  Judging  from  the  present 
record  it  might  have  been  considerably  better.  Therefore,  it  asess  wise  to 
piece  little  value  upon  the  retraining  data  of  P-81,  unless  it  is  considered 
representative  of  performance  under  adverse  conditions  not  typical  of  tho 
conditions  under  which  the  others  performed.  These  ©vents  also  may  have  had 
consequences  for  the  retraining  performance  of  P-82  and  P-83  for  the  earn® 
reasons  mentioned  above.  However,  it  is  surmised  that  such  effects*  if  any, 
were  slight — again  for  parallel  reasons. 

Finally*  still  with  respect  to  the  conduct  of  the  test,  at  least  two 
worthwhile  issues  can  be  raised  regarding  retraining.  In  tho  first  place* 
what  are  the  consequences  for  interpretation  of  tho  different  amount  a  of  train¬ 
ing  given  the  crews  in  the  two  study  phases?  That  crows  0-8  asd  0-13  were 
given  Each  more  retraining  than  0-4  and  0-9  is,  clearly,  a  Eattar  of  design 
attributable  to  the  earlier  experience  with  0-4  and  0-9  and  the  result  lag 
suspicion  that  they  were  net  fully  trained.  Had  0-4  and  0-9  been  given  more 
retraining  it  would  bo  possible  to  fully  compare  them  with  C-8  and  0-13  and 
perhaps  more  fully  assess  the  limitations  in  their  original  Skill.  However, 
judging  from  ihs  data  at  hand,  0-4  and  0-9  probably  would  not  hare  achieved 
their  skill  potential  ovon  if  thoy  bad  received  tk®  two  additional  days  of 
training  given  the  others.  Therefore,  it  would  ©till  have  been  difficult  to 
quantify,  even  after  tho  fact,  their  original  level  of  skill  for  the  purpose 
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of  relating  this  to  skill  retention  and#  in  turn#  increasing  the  value  of  the 
study.  Hor®  retraining,  unless  sufficient  for  them  to  reach  thsir  skill  poten¬ 
tial,  would  not  have  rendered  their  data  catch  sore  useful. 

Th®  second  concern  about  retraining  has  to  do  with  th©  way  in  which  the 
retraining  was  given  to  0-8  and  0-13*  In  particular,  of  what  ccasoguenee  is  it 
that  in  most  oases  the  individual  performed  fair  trials  cat  a  given  phase  in  rapid 
sequence  and  then,  being  replaced  by  another  pilot,  vent  to  another  position  or 
to  a  rest  period?  Certainly  it  night  be  supposed  that  sc®a  advantage  night  have 
been  gained  from  frequent  repetition  of  the  seas  thing  with  fairly  aapl®  rent 
opportunities  interspersed*  Ca  the  other  hand  sees  disadvantage  night  have  accrued 
fro®  the  disrupt  ion  of  the  mission  sequence  and  the  need  to  wars  up  to  porforesaoa 
of  th®  phase  eokeduled.  In  general,  however,  it  is  supposed  that  any  such 
effects  war®  sufficiently  slight  and  so  variably  distributed  throughout  retrain¬ 
ing  as  to  little  influence  th®  estimation  of  capability  of  primary  oonosra  ia 
this  study. 

In  fact,  a  similar  view  sseas  generally  appropriate  with  respect  to  all  th© 
factors  discussed  relating  to  the  conduct  of  the  test*  Shore  is  little  question 
bat  what  the 'various  factors  mentioned  operated  to  drive  th®  performance  up  or 
down  on  occasion,  However,  with  the  exception  of  P-8l*s  rotmining  performance, 
it  is  unlikely  that  these  effects  were  of  any  consequence.  But,  if  th@r©  was 
any  overall  net  effect  of  then  it  was  probably  perforjnanec-dsgrading  rather  than 
perfornvance-onhancing. 

Task  Set  of  Tost  Personnel.  One  other  feature  of  th®  data  collection 
procedures,  which  is  perhaps  of  greater  significance  than  all  the  others,  con¬ 
cerns  the  instructions  given  the  test  personnel  about  the  nature  of  the  test  and 
what  was  expected  of  them.  Hi  daily  oirouastanees  as  well  as  in  formal  laboratory 
experimentation  the  way  in  which  a  man  views  or  defines  his  task  (that  is,  his 
task  goals)  has  a  great  influence  on  how  he  will  perform  and  how  he  will  divide 
his  attention  among  ths  several  aspects  of  the  task.  Consequently  and  because 
the  tasks  involved  in  this  test  ware  complex  ©nd  multifaceted  it  is  of  great 
relevance  for  any  interpretation  of  the  data  to  ascertain  th®  nature  of  ths 
task  goals  held  by  the  test  personnel  as  they  performed  at  various  times.  In 
particular,  it  is  important  to  determine  whether  these  goals  may  have  shifted 
from  training  to  tost  mission  to  retraining  with  resulting  differential  effects 
upon  performance. 

Now,  for  the  fullest  sort  of  information  on  how  the  pilots  viewed  their 
tasks,  reooursa  would  be  necessary  to  soma  sort  of  interview  and/or  questionnaire 
technique  intended  to  elucidate  their  thinking  about  then  in  depth.  And  to  some 
extent  this  was  accomplished  with  respect  to  task  particulars,  as  reported  by 
Orodsky  ot  al  ( 1966b),  but  it  was  not  accomplished  with  respect  to  general  goals. 

To  have  attempted  to  obtain  information  on  general  goals,  except  by  perhaps  the 
most  subtle  technique,  would  have  engendered  antagonism  on  the  pilots'  part. 

Their  natural  reopens®  would  have  bean  in  the  vein,  "¥@11,  of  ©corse,  I  did  what 
you  asked  as  to  do."  furthermore,  individuals  are  often  unable  to  explicitly 
formulate  th©  goals  implicit  in  their  activities. 

nevertheless,  in  th®  absence  of  formally  taken  direct  expressions  on 
general  goals,  It  is  possible  to  cafe®  sees  reasonably  well-founded  conjectures 
about  thorn  on  th®  basis  of  th©  infractions  given  the  test  personnel  and  their 
informal  comments  to  th©  test  staff.  On  these  grounds  a  rather  clear  distine- 
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tion  can  be  drawn  between  the  probable  approaches  of  pilots  la  0-8  and  0*13  to 
their  tasks  in  the  test  mission  and  in  retraining  versus  their  goals  in  training* 

A  useful  distinction  also  can  be  drawn  between  the  handling  of  the  two  sets  of 
crews  in  the  two  phases  of  the  study# 

Thus*  the  two  crews  of  the  earlier  phase  (0*4  and  0*9)  received  rather 
general  encouragement  to  perform  well  in  training  and  subsequently  in  test  and 
restraining.  Specific  goal  criteria  in  flight  control  parameters  were  not  iden¬ 
tified  for  these  crews  as  guides  to  their  practice  and  performance,  nor  w»re 
they  regularly  provided  precise  information  about  their  performance.  No  doubt 
they  did  operate  with  some  awareness  of  what  would  be  considered  good  and  had 
performance  but  these  were  never  precisely  defined* 

In  contrast,  throughout  their  training  and  during  the  testing  sequence 
0*8  and  0-13  performed  against  known  flight  control  goals  for  each  phase,  such 
as  were  listed  in  table  V#  (The  actual  criteria  used  included  a  few  more  than 
are  represented  in  table  V. )  Pilots  of  these  crews  were  also  encouraged  to  do 
as  well  as  possible,  but  it  was  thus  additionally  evident  exactly  what  was  con¬ 
sidered  good  and  bad  performance.  Furthermore,  daily  displaying  to  each  pilot 
his  previous  day’s  performance  by  means  of  continuous  graphs,  with  criteria 
indicated,  served  further  to  emphasise  the  urgency  of  meeting  the  criteria  and 
to  bring  about  close  attention  to  rate  of  progress.  In  all  likelihood  these 
procedural  differences  in  the  way  in  which  the  two  sets  of  crews  were  handled 
resulted  in  a  more  favorable  learning  situation  for  0-8  and  0-13,  at  least  up 
to  the  point  of  consistent  capability  to  meet  criteria  in  repeated  trials.  This 
effect,  in  addition  to  sheer  differences  in  amount  of  training  received,  may  be 
largely  responsible  for  their  greater  skill  at  the  end  of  training.  Py  the  end 
of  training  the  pilots  of  0-8  and  0-13  had  achieved  sufficient  skill  to  regularly 
meet  the  oriteris  in  most  parameters,  as  table  XXV,  appendix  III,  shows.  Clearly 
then,  they  had  met  the  requirements  of  the  original  reliability  study  as  defined 
for  them  and  it  was  with  this  recall  of  their  own  former  capability  that  they 
approached  the  skill  retention  test. 

However,  in  the  course  of  the  initial  briefing  to  the  pilots  the  day  be¬ 
fore  skill  retention  testing  began,  a  strong  attempt  was  made  to  redefine  the 
task  goals  for  0-8  and  0-13  for  the  skill  retention  work.  In  these  briefings 
the  pilots  of  both  crews  were  asked  to  seek  the  very  bast  performance  possible 
as  consistently  as  possible,  rather  than  to  merely  strive  to  meet  the  stated 
task  criteria.  By  means  of  blackboard  illustrations,  they  were  shown  how  not 
only  typical  performance  level  but  also  variability  in  performance  would  be 
employed  to  compute  reliability  figures  for  their  performances  in  the  several 
parameters.  And,  they  were  told  that  these  reliability  figures  would  be  taken 
as  the  primary  indication  of  their  capability. 

These  new  instructions  concerning  task  goals  represented  for  the  pilots  a 
seemingly  radical  departure  flora  previous  instructions  as  well  as  from  accepted 
practice  in  aircraft  operation.  As  a  result  they  naturally  responded  with  some 
degree  of  confusion  and  requests  for  clarification.  Clarification  was  provided 
until  no  more  questions  were  forthcoming,  but  even  then  it  seemed  evident  to  the 
study  staff  that,  although  the  pilots  accepted  the  new  task  goals,  they  were  not 
comfortable  with  them.  Subsequent  comments  exchanged  during  the  data  collection, 
on  review  of  test  records,  etc.,  tended  to  confirm  this  supposition  and  to  suggest 
that  the  general  attitude  persisted  throughout  the  study.  Nevertheless,  their 
comments  also  suggested  th&t  they  wore  attempting  to  carry  cut  instructions j  i.e., 
to  turn  in  the  best  possible  performance  as  defined  in  the  initial  briefing. 


* 


To  the  art  eat  the  pilots  actually  adopted  it,  this  r©ari«ststiea  to  the 
aisaicn  tasks,  as  contrasted  with  the  approach  in  training,  probably  has  signif¬ 
icant  implications  for  the  iniorpretat ion  of  the  dais,  %ila  earlier  they 
Barely  had  to  m&t  the  criteria,  now  they  wars  to  do  that  aid,  further,  to  exr 
coed  thsa  as  such  m&  as  consistently  as  possible.  Certainly,  with  these  nm 
goal®  hofcrc  thsas  they  would  naturally  tend  to  rsassaine  the  ocrcrel  aepsets  of 
their  perforaenoe  in  ©sob  phase,  looking  for  ways  of  adjusting  it  hy  rsaaowed 
or  speoial  attention  to  one  aspect  or  another,  particularly  those  in  which  they 
typically  performed  less  well*  They  also  would  to  exacted  to  e^psriesat  with 
different  operating  strategics  for  accomplishing  the  s»ra  ccBpleac  phases,  gueh 
os  braking  end  hover  and  earth  entry,  and  esae  pilot  did  xasintsKticnally  ecafira 
this  expectation,  about  his  am  perfoasaaaos. 

The  net  effects  of  these  actions  on  reoordM  porfozfiase®  would  bn  a  pre¬ 
dictable  gain  in  retraining  so  far  as  the  criteria  ars  ©oaoorasd.  So@®wr,  the 
gain  would  not  necessarily  be  regular  or  great  for  in  seeking  and  shifting  to 
new  operational  strategies  so®®  interim  loss  would  be  expected*  Perfection  of 
a  new  strategy  would  require  at  least  a  few  trials*  On  the  other  hand,  the 
effects  of  those  sssao  actions  upon  perfor&aaea  as  interpreted  by  reference  to 
former  level©  of  estimated  high  reliability  would  be  quite  different.  for9 
particularly  the  overall  phase  and  aission  reliabilities  {j?  and  &)  would  bs 
adversely  affected  by  the  great  preponderance  of  ©ospeaeatory  shifts.  To  gain 
with  respect  to  the  criterion  in  a  Isas  adequately  performed  parajsster  would 
be  to  gain  also  with  respect  to  former  p  m  level,  to  a  probability  >.950. 

Bit,  to  at  the  earn©  tine  sacrifice  soe©  performance  in  another  paraisat or,  since 
the  pilots  were  generally  capable  beyond  the  .999  level  of  seating  the  criterion, 
often  would  result  in  no  detected  change  in  performance  with  faspeot  to  criterion, 
but  always  considerable  lose  balow  .^0  with  respect  to  former  level*  The  nature 
of  this  differential  affect  will  be  axaainad  in  sor®  detail  later.  For  ik@ 
present  it  seroly  serves  to  characterise  the  important  indications  of  the  charge 
in  task  goals,  which  presumably  occurred. 


Beyond  the  procedures  used  in  data  collection,  there  ar©  still  th©  specific 
analytic  procedures  used  to  summarise  and  organise  the  data  obtained;  The  choice 
ffisong  these,  too,  can  have  a  strong  bearing  upon  the  conclusions  roaohod  for 
they  influence  directly  the  nature  of  the  analytic  results  obtained.  This  is 
particularly  true  of  tho  present  study  which,  being  in  sceo  rays  quite  unlike 
traditional  psychological  research  in  goals  and  design,  required  the  use  of  novel 
analytic  procedures  to  render  the  data  Boaaingful.  The  details  of  tho  analytic 
approach  are  given  in  appendix  II,  rather  than  being  ouaborsocoly  included  within 
the  body  of  this  report,  and  no  repetition  of  thoa  will  be  given  here.  However, 
there  arc  certain  particulars  which,  because  of  their  possibly  groat  influence 
on  the  results  obtained,  should  bo  explicitly  recognised. 

Sailing  Moini sm.»  Among  these  important  analytic  details  ar®  tho  decis¬ 
ions  kM©  regarding  the  sailing  of  data  available.  In  the  usual  concern  for 
caapliag,  the  interest  is  in  the  set  of  test  personnel  eaployod  and  tho  manner 
in  which  this  sot  is  selected  from  tho  noro  ino!us ivo  population  which  they  aro 
supposed  to  represent.  This  usual  concern  is  applicable  to  tho  present  study 
as  troll  and  will  be  touched  on  in  a  following  portion  of  this  section.  Sere, 
the  interest  is  in  a  different  sampling  problem  involving  tho  selection  cf  a 
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limited  portion  of  one  individual’s  date  to  best  represent  that  individual, 
Fundssesrtaliy,  this  problem  occurs  largely  because  in  such  date  as  these 
characteristic  trends  are  eapectedj  that  is,  it  is  anticipated  that  learning 
or  forgetting  sill  occur  with  consequent  changes  in  performance.  Thus,  in  the 
language  of  mathematical  statistics,  ons  is  dealing  with  a  noastationary  process. 
Sow  then  can  one  hope  to  arrive  at  an  estisate  of  performance  level  at  certain 
times  from  data  series  which  met  he  assessed  changing  in  one  or  more  aspects? 
Beyond  the  nuances  of  this  problem  and  more  to  the  point  of  the  present  discuss¬ 
ion,  saspling  for  the  purposes  of  this  analysis  was  done  in  a  certain  way  and 
this  may  have  influenced  the  results  in  a  predictable  fashion. 

Thus,  in  the  first  place,  with  but  one  exception,  whenever  there  wore 
sufficient  data,  the  siise  of  the  performance  sample  taken  waa  always  four.  Some 
retraining  phases  of  0-4  and  0-5  are  ths  exceptions.  This  was  done  because  four 
trials  were  estimated  to  be  an  optimum  compromise  between  inaccuracies  arising 
from  the  inclusion  of  trends  and  from  error  in  parameter  estimates  from  small 
saasplas.  Obviously  a  larger  sample  sise,  say  five  or  six,  would  have  resulted 
in  less  estimation  error  if  the  date  series  was  stable  but  not  necessarily  so 
if  tho  series  was  still  in  the  process  of  change.  Whether  four  was  the  optimum 
is  naturally  debatable,  but  there  is  no  doubt  estimations  deriving  fro®  a  sample 
of  four  vary  considerably,  in  accordance  with  the  relatively  greater  wei$it 
accorded  extreme  values  in  the  small  sample  Kid  are  occasionally  in  considerable 
error  (bo  far  as  the  mean  of  repeated  such  samples  is  concerned) ®  It  follows 
then  that  particular  values  obtained  for  p  mast  be  taken  with  qualification, 
especially  when  they  deviate  extremely,  for  they  are  overinf lusnce d  by  an  occa¬ 
sional  performance  extreme.  This  would  also  be  true  to  an  even  greater  degree 
of  p  values  taken  jointly  (i,e«,  $)  for  the  joint  product  is  strongly  influenced 
by  inequalities  among  the  elements  involved.  Chi  the  other  hand  p  values  taken 
cumulatively  should  represent  quite  accurate  estimates,  since  any  errors  of 
estimation  are  presumably  randan  ana  so  tend  to  cancel  out. 

Besides  sample  size,  one  other  aspect  of  the  data  sampling  of  special 
interest  is  the  particular  set  of  trials  selected  or  the  portion  of  the  data 
taken.  In  having  chosen  a  relatively  small  sample  sise  and  also  elected  to 
limit  attention  to  only  the  last  four  training  trials  it  is  quite  possible  that, 
because  of  a  low  extreme  in  the  trials  selected,  the  individual’s  capability 
was  underestimated,  (in  fact,  correct  ion  in  two  such  oircuastanoes  ©eessed 
necessary,  as  already  described. )  There  is  no  reason  to  suppose  that  the  last 
four  trials  were  the  individual’s  bast,  particularly  if  he  was  no  longer  learn¬ 
ing  rapidly.  Thus,  it  might  be  suggested  alternatively  that  the  best  four  train¬ 
ing  trials  should  have  been  used  as  truly  representing  the  individual’s  greatest 
attainment.  Certainly  this  choice  is  a  debatable  one  for  taken  at  face  value 
the  best  four  trials  indeed  must  be  the  best  performance.  On  ths  contrary,  to 
take  the  best  four  trials  might  be  to  overestimate  the  capability  (mindful  of 
the  sample  size),  whereas  use  of  the  last  four  would  result  in  random  over- 
and  underestimation  and  no  overall  bias.  Also,  from  the  practical  point  of  view, 
finding  the  best  four  trials  takes  more  work  because  more  data  must  be  analyzed. 

Nevertheless,  whatever  the  ultimate  resolution  of  this  issue  may  bs,  tho 
last  four  trials  wore  taken  as  the  reference  with  the  probable  result  that  on 
some  oooasions  the  estimates  derived  ever-  or  underestimated  the  "true”  capability. 
A  casual  check  by  reviewing  the  results  of  an  alternative  weighted  Z-soore 
analysis  performed  on  the  same  data  did  reveal  a  number  of  cases  involving  0-8 
and  0-13  in  whioh  a  trial  blook  other  than  the  last  apparently  was  superior. 
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But  on  closer  ©raialnation  of  these  by  reference  to  comparable  phase  probabilities 
it  was  found  that  there  are  relatively  few  instances  in  uhioh  the  actual  superior- 
ity  ever  ths  last  trial  block  is  considerable.  For  the  0-8  pilots  ths  shifts 
in  p  sad  $  ^aro  as  follows*  P-8l  in  SIX),  ,919  to  .99 8  and  .75?  to  .995}  P-82 
in  SBO,  .953  to  .987  and  .860  to  .950,  end  in  LA,  .80 6  to  >.999}  P-83  in  TL1, 

•954  to  .997  sad  .908  to  .994*  For  the  0-13  pilots  th@  shifts  in  p  and  $  Kero* 
P-131  in  JH,  .980  to  >,999  and  .920  to  >.999}  P-132  in  330,  ,739  to  .978  and 
.370  to  .935,  sad  in  2ok,  .939  to’  .980  and  .819  to  .940?  P-133  in  U,  .557  to 
.865,  and  in  Bokf  ,942  to  >.999  and  .827  to  >.999*  The  affects  of  substituting 
those  new  estimates  of  reliability  in  certain  phases  on  estimated  mission  reli¬ 
ability  say  bo  seen  in  table  XVIII,  which  shows  both  ths  old  and  nav  p,  and  p 
values  for  individuals  and  crews.  Obviously,  ths  pilots*  original  capabilities 
(particularly  P-132  and  P-133)  are  estimated  to  bs  raich  higher  when  the  bast 
four-trial  blook  performance  is  taken  as  ths  reference.  But,  it  is  still  dabat- 
able  whether  this  procedure  gives  a  nor®  valid  estimate.  Perhaps  the  most 
accurate  value,  if  available,  would  be  found  to  compromise  between  the  two  of 
table  XVI II. 


TABLE  ran 

Mission  Beliability  of  0-8  and  0-13  Per  Criteria  in  Training 
as  Estimated  from  the  Last  Four-  and  the  Best  l^jcr-Trial  Block  * 

Measure 

25  *7*  t 

P  P  P 


Crew/Pilot 

last  4 

* 

best  4 

last  4 

A 

bast  4 

last  4 

4f 

best  4 

0-3  1 

•  990 

•999 

,971 

•997 

•741 

,974 

2 

.973 

•9  98 

.961 

•994 

•683 

•947 

3 

•991 

.996 

.978 

,988 

.817 

.894 

all 

•984 

.998 

.970 

•993 

.747 

.938 

0-13  1 

•96  7 

.969 

♦946 

•955 

.582 

.633 

2 

.961 

.993 

•907 

.983 

.294 

.854 

3 

•932 

.973 

.907 

•960 

•361 

.679 

all 

.953 

.978 

.920 

.966 

.412 

.722 

*  four-trial  blocks  t'oen  as  superior  to  the  last  four-trial  blocks  and 
substituted  for  them  are  only  those  v&ich  a  weighted  Z-score  analysis 
suggested  would  be  greater  and  for  which  p  for  that  mission  phase  was 
subsequently  found  to  be  at  least  .008  greater 


Furthermore,  even  if  the  best  four-trial  blook  in  training  is  taken  as 
a  reference  the  consequences  for  the  analysis  of  test  mission  capability  to 
meet  criteria  are  slight.  For  0-8  only  SBO  estimates  are  aff ec ted— P-8l  shift' 
ing  from  .713  to  .662  and  .138  to  <.001,  and  P-82  shifting  from  .980  to  .99° 
and  .939  to  .989  in  likelihood  of  meeting  criteria  (g  and  p).  The  result  in 
overall  reliability  for  P-8l  is  a  shift  from  .931  to  .925,  .827  to  .812,  and 
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.048  to  <.001  in  f> ,  $9  and  $.  The  overall  shifts  for  P-82  are  .963  to  .958, 

.920  to  .929,  and  .395  to  .431.  In  0-13,  P-131  shifts  from  .294  to  .250  in  p 
on  BH  (no  change  in  j>),  with  a  resulting  shift  in  f  from  .899  to  .894  (p  and  f> 
unchanged).  P-132  shifts  (?  and  $)  from  .893  to  .999  and  .695  to  .997  in  S2JQ 
and  from  ,992  to  .994  and  .975  to  .998  in  Itek  with  changes  in  overall  reli¬ 
ability  (P,  p,  and  p)  from  .982  to  ,995»  .958  to  .994  and  ,643  to  .945*  P-133 
shifts  from  .716  to  ,885  in  IPA  and  from  .951  to  >#999  and  .853  to  .$>99  in  Jtok 
with  changes  in  overall  reliability  from  .954  to  .978,  .927  to  .962,  and  .481 
to  .697.  Thus,  as  noted  earlier,  it  appears  that  extroaa  values  in  the  last 
four  training  trials  are  of  concern  only  in  the  analysis  of  the  test  perform¬ 
ance  of  P-132  and  P-133.  If  these  extreme  values  are  considered  spurious  then 
capability  to  meet  criteria  in  test  must  be  considered  somewhat  underestimated 
for  both  P-132  and  P-133 — perhaps  the  most  accurate  estimate  being  some  compro¬ 
mise  value.  Parallel  roanalysie  of  test  mission  capability  to  meat  former  p#^c 
levels,  with  the  best  four-trial  blocks  in  training  used  in  computing  those 
levels,  results  in  probabilities  that  are  never  greater  Mid  are  generally  less 
than  those  originally  obtained.  Although  these  reductions  in  probability  are 
small  and  of  little  consequence,  again  the  probabilities  for  P-132  and  P-133 
are  found  most  affected.  The  values  for  P-132  shift  from  .871  to  .831  in 
from  .731  to  .627  in  f>,  and  from  .004  to  <.001  in  f>.  Those  for  P-133  shift 
from  .805  to  .758,  from  .570  to  .49I»  and  ftom  .001  to  <.001.  Accordingly, 
the  capabilities  of  P-132  and  P-133  to  meet  former  p  levels  may  be  con¬ 
sidered  slightly  overestimated  to  the  extent  that  extreme  values  recorded  for 
them  at  the  end  of  training  are  spurious. 

Comparison  of  best  retraining  performance  with  best  training  performance 
still  shows  a  gain  in  capability  to  meet  criteria  for  four  of  the  six  pilots. 

By  this  comparison  P-8l  never  did  quite  as  well  in  retraining  as  in  training 
(•991  vs  .998  in  p  and  .978  vs  .995  in  p),  probably  for  the  reasons  already 
discussed.  Also,  P-133  did  not  demonstrate  his  original  capability  to  mset 
criteria  (.991  vs  .999  in  p,  .964  vs  .996  in  p,  and  .789  vs  .976  in  p).  The 
cumulative  probabilities  for  crews  still  indicate  a  net  gain  in  capability  to 
meet  criteria  in  retraining.  The  parallel  analysis  of  greatest  demonstrated 
capability  in  retraining  to  meet  former  p  nc  levels,  in  making  reference  to 
occasionally  superior  phase  performance,  can  never  result  in  probabilities 
greater  than  those  based  on  the  last  trial-block  reference. 

Thus,  it  is  concluded  that  the  use  of  the  last  four  training  trials  as  a 
basic  reference  had  little  affect  upon  the  main  outcomes  of  the  analysis  of  0-8 
and  0-13  performance.  It  apparently  resulted  in  underestimation  of  original 
capability  to  meet  oriteria,  of  much  consequence  only  for  P-132  and  P-133.  In 
turn,  the  capabilities  of  P-132  and  P-133  for  meeting  the  oriteria  in  the  test 
mission  appear  somewhat  underestimated,  whereas  their  capabilities  to  meet  former 
p  Ac  levels  appear  slightly  overestimated.  However,  the  critical  estimations  on 
test  performance  of  P-81  and  P-131,  on  which  the  main  conclusions  must  rest,  are 
virtually  unaffected.  The  outcomes  of  comparisons  involving  retraining  also 
remain  essentially  unchanged,  even  though  use  of  the  last  four  training  trials 
may  have  introduced  a  slight  overestimation  of  capability  in  retraining, 

A  similar  but  still  more  complicated  dilemma  exists  with  the  selection  of 
a  four-tidal  block  to  best  represent  capability  in  retraining.  Bather  obviously 
the  first  trial  block  would  not  be  the  best  if  relearning  or  new  learning  is 
assumed.  But,  the  last  one  might  not  ba  the  bast  either,  particularly  if  there 
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i a  seas  evi&mcQ  at  or  basis  for  a sensing  eantradiotory  tread  effects  f!roa  tbs 
presses  of  eose  factor  other  then  learning*  tJhfostun&tely,  the  presence  of 
just  euch  an  effect  does  seen  Indicated  Igr  the  retraining  data  for  $-8  and  0~135 
and  staff  observation  suggest®  a  contiming  dsganenatisa  in  tbs  task  motivation 
of  the  test  persons©!  after  the  first  day  of  rstr&iniag  as  tbs  acoinm table  fac¬ 
tor,  TheTofcTi}-  it  boss®  seat  reasonable  in  this  instates  to  taka  the  bast  four 
trials  as  sect  *  apresent&tive  of  individual  c^sability— tha  esticstea  derived 
frosa  then  being  inflated  by  sampling  biaas  bat  deflated  by  reduced  aotivaiisa 
in  roughly  a  compensatory  fashion.  It  in  a  natter  of  conjecture  just  how  truly 
representative  the  ©ctisatoa  so  obtained  are*  feat  it  is  believed  they  are  prob¬ 
ably  reasonably  close  (say  within  ,01  in  p). 

However,  the  choice  of  th®  best  four  trials  does  not  exhaust  the  ratifica¬ 
tions  of  the  sampling  pnoblesa  with  respect  to  rot  raining  data.  It  was  still 
necessary  to  decide  whether  the  best  four  trials  with  rsspeet  to  p  or  t  should 
be  used  and  whether  these  should  b©  with  respect  to  criteria  (g0  or  p0}  or 
whether  they  should  bo  with  respect  to  the  former  attainment  in  training  ($5*95 
or  $.95).  For  the  main  analysis  of  results,  comparing  the  several  levels  of 
perfornanc©  exhibited  throughout  the  azpcricsnt  (see  table  Xltf),  this  wao  resolved 
by  taking  $0  for  th©  critericn-rofsrenced  analysis  and  for  the  tomar  level- 
reforenood  analysis.  The  p  was  oonaidarod  superior  to  f>  aa  bast  or  sore  sen¬ 
sitively  reflecting  the  overall  capability  in  any  phase. 

This  solution  does  bring  about  en  analytic  dilessa  stealing  directly  froa 
th©  fact  thL  the  boat  four-trial  block  indicated  by  £05  performance  is  eor© 
than  a  few  tines  not  the  bast  trial  block  indicated  by  pc  performance.  In  other 
words,  two  ecsewhat  different  sets  of  retraining  trials  are  used  to  represent 
retraining  capability.  This  circumstance  might  surely  have  implications  for  the 
comparison  of  gains/losses  in  retraining  over  original  levels  as  derived  from 
oriterion-rofercnced  versus  former  level-referenced  probabilities. 

Such  a  comparison  of  the  performances  of  0-8  raid  0-13  is  summarized  in 
table  XIX.  Clearly,  aa  stated  in  the  final  portion  of  the  presentation  of 
results,  so  far  as  th©  criteria  go  thaeo  crows  gained,  but  so  far  aa  former  p#^ 
levs!  is  esnesmsd  they  lost  in  retraining  over  endrof— training  capability. 

The  question  whioh  naturally  follows  is  whether  this  contradictory  result  is 
to  b©  accepted  as  plausible  and  solely  the  result  of  th©  shifts  in  task  sot 
already  described,  or  whother  it  nay  not  also  b©  partly  th©  consequence  of  the 
selection  of  best  four-trial  block  in  retraining.  Specifically,  what  would  bo 
th©  alternative  results  if  p  95  war©  computed  on  th©  basis  of  th©  bast  oriterion- 
roforencod  block  when  estimated  capability  to  meet  criteria  is  nsudasal  (becom¬ 
ing  i>.95/o)7  Similarly,  what  would  happen  if  p0  war©  computed  on  th©  basis  of 
the  boat  po95  lavol-rafercncod  block  whan  capability  to  achieve  former 
levels  in  individual  paraae  ;ers  is  tsaxlBal  (becoming  P0/.95)? 

Several  deductions  about  the  relative  orders  of  magnitude  of  the  alterna¬ 
tive  p  values  and  the  croas-coapariotxa  of  thorn  can  be  made.  In  the  first  plae® 
pG/0  must  ba  greater  than  Pc/.93»  because  Pc/. 95  i®  based  on  at  least  some 
ssallor  p  values.  Sinoo  pn/0  in  retraining  is  known  to  represent  a  gain  over 
training,  Po/,95  East  ro®4  to  show  either  less  gain,  no  difference,  or 
possibly  a  loss.  Similarly  P.95/.95  ^ust  b©  greater  than  V,<fi/Q  bo causa  th© 
latter  is  based  on  boss  ssallor  p  values.  Further,  since  P-95/  95  has  boon 
found  to  represent  a  loss,  P,$5/o  EU3t  bo  found  to  represent  still  greater  loss. 
Thus,  in  the  comparison  of  gain/loss  of  P0/,95  and  95/^95  the  differences  must 
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percentage  gain  or  loss  indicated  in  parentheses 

TAJETS  XX 

Change  in  Mission  Reliability  of  0-8  and  0-13  frosa  Training  to  Beat 
Fonr-Trial  Block  in  Retraining  per  Alternate  Reference  Measures  * 
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bo  lass  than  that  apparent  in  table  XVII,  while  in  the  comparison  of  gain/ loss 
of  P.95/0  end  pft/0  tas  MJtf&rmz sss  mst  h©  still  greater.  Furthsresor®,  th© 
probabilities  obtained  ehculd  be  ordered  f&ca  most  gain  to  aost  loss  as  follows? 

p°/o>  po/.95  ?  p.95/.$5  >P.95/o 

'Those  deductions  say  bo  confirmed,  by  cros^-eosparing  the  tabulations  of  p 
computations  based  ea  P.95/0  p0/  ar  given  in  table  3S  with  those  of  table 

XIX.  Which,  then,  of  ih©  three  possible  pairs  is  the  most  valid  and  useful  one 
for  representing  retraining  capability  or  should  all,  perhaps,  be  used  la  concert 
to  provide  a  fuller  understanding  of  the  studs'  results? 

Tha  latter  view  seess  unquestionably  the  more  appropriate  on®.  Clearly 
p0/c  comparisons,  as  representative  of  capability  to  meet  the  task  goals  under¬ 
lying  all  performance,  are  of  primary  concern.  The  retraining  over  training 
gain  in  then  must  be  taken  at  face  value  as  indicative  of  genuine  improvement. 

In  addition,  that  the  oreweeabers  showed  largo  loseos  as  in  the  V,^/0  compar¬ 
isons,  while  simultaneously  gaining  with  respect  to  criteria,  suggests  a  definite 
change  in  approach  to  the  task.  That  this  change  cannot  bo  considered  to  any 
groat  extent  an  artifact  of  the  retraining  trial  sample  used  is  demonstrated  by 
the  similarly  low  P,55/.  95  values.  Evidently,  although  tbs  problos  of  sampling 
In  retraining  introduce  complexities  in  analysis  it  doss  not  poa©  particular 
difficulty  for  interpretation.  Certainly  the  differences  in  results  concerning 
retraining  would  bs>  to  only  a  alight  extent,  if  any,  tha  consequences  of  the 
sampling  used. 

33>gJ*Mia&a^faafi^  'Tacitly  aasumad  in  the  fore¬ 

going  discussion  of  the  sampling  problem  is  that  the  sample  data  are  to  be 
interpreted  in  terms  of  some  statistical  model.  In  this  iastancs,  the  modal 
adopted  is  the  very  commonly  used  normal  density  function.  Thus,  in  using  this 
model  it  is  being  assumed  that  the  mean,  standard  deviation,  raid  other  parameters 
of  a  hypothetical  sampling  distribution  of  the  data  samples  under  examination 
approximate  those  of  a  normal  distribution,  if  this  assumption  is  acceptable 
then  the  interpretation  of  the  data  sample  in  terms  of  the  theoretical  normal 
distribution  leads  to  no  systematic  bias.  On  the  other  hand,  if  it  can  be 
shown,  or  there  is  a  reasonable  basis  for  assuming  that  the  sample  data  depart 
from  the  normal  in  certain  ways,  then  the  possible  biases  introduced  mast  be 
considered.  There  are  at  least  three  specif io  questions  concerning  use  of  the 
normal  distribution  which  should  be  discussed. 

First,  although  many  human  performance  measures  have  been  found  to 
approximate  normal  distributions  this  is  not  invariably  true  and  the  question 
may  properly  be  raised  as  to  whether  the  measures  used  in  this  study  did  dis¬ 
tribute  normally.  If  performance  had  at  any  time  stabilized  (on  some  acceptable 
criterion)  over  an  appreciable  number  of  trials  a  rough  answer  to  the  question 
might  have  been  obtained  by  tha  use  of  tests  for  normality.  But,  because  trends 
were  likely  and  a  four-trial  sample  was  chosen  as  the  analytic  bass  standard 
tests  for  normality  ware  not  feasible.  Accordingly,  the  question  must  bo 
answered  simply  (if  at  all)  on  the  basis  of  an  examination  of  the  data  and 
awareness  of  measurement  factors  commonly  resulting  in  nonnormal  distributions. 
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Eeview  of  tbs  data  did  not,  however,  disclose  evidence  of  sufficiently 
frequent  departures  from  normality  as  to  give  pause  for  eardous  concern*  There 
wsre,  of  course,  a  few  instances  in  which  the  last  four  training  trials  did  not 
seem  appropriately  variable  or  seemed  skewed,  but  this  is  to  la  expected  with 
eo  small  a  sample.  Certainly  the  few  instances  can  hardly  be  takas  as  proof 
that  the  normal  distribution  i?aa  not  generally  applicable.  Even  the  critical 
velocity  cutoff  error,  treated  as  it  was  without  regard  to  sign,  did  not  clearly 
exhibit  the  degree  of  asymmetry  which  might  be  expected*  Apparently,  that 
there  was  almost  always  some  such  error  Kid  that  it  tended  to  be  biased  the 
same  direction  from  zero  resulted  in  this  measure  bearing  a  reasonable  semblance 
of  a  normal  distribution  only  moderately  truncated  by  the  zero  bound.  The  other 
error  measures  (displacement  and  displacement  rate)  which  also  might  ba  supposed 
to  present  a  similar  problem  apparently  were  sufficiently  large  as  not  to  present 
serious  truncation  or  asymmetry  either,  fti  tka  other  hand,  it  is  not  being 
contended  that  sampling  distributions  of  these  measures  (and  perhaps  certain  of 
the  others)  would  be  normal-only  that  they  would  be  sufficiently  ©lose  to 
normal  that  the  application  of  the  normal  model  is  feasible* 

Apart  from  these  possible  gross  departures  from  the  normal  as  a  model, 
there  is  the  obvious  arbitrary  use  of  it  to  arrive  at  probabilities,  as  simple 
indicants  of  performance  capability  in  the  measured  flight  control  parameters. 

The  chosen  sample  aim  was  indeed  small — much  smaller  than  is  commonly  .inter- 
preted  by  reference  to  the  values  of  the  normal  density  function.  With  such 
small  samples  of  an  otherwise  normal  variate,  the  3tudent  t.  distribution  is  the 
traditionally-recommended  model  for  a  closer  approximation  to  the  true  prob¬ 
abilities.  And,  unquestionably,  it  would  have  provided  more  exact  values  for 
these  data  as  well. 

However,  the  use  of  the  normal  rather  than  the  t,  distribution  seemed 
justified  in  this  instance  on  two  bases.  These  are  the  ready  availability  of 
extensive  normal  probability  tables  showing  values  to  the  nearest  thousandth 
and  the  nature  of  the  interest  in  the  data*  As  has  already  been  noted  and  will 
be  made  still  more  evident,  the  absolute  levels  of  the  probabilities  reported 
would,  in  any  case,  have  to  bo  taken  with  considerable  qualification  and,  further¬ 
more,  the  primary  interest  in  the  study  data  is  in  performance  changes  or  differ¬ 
ences  rather  than  in  absolute  level  of  reliability.  Therefore,  the  inconvenience 
of  using  the  jt  distribution  did  not  seem  warranted.  Inaccuracies  in  p  arising 
from  this  procedure  would  tend  to  run  parallel  in  any  two  sets  of  performance 
and  so  should  not  significantly  affect  the  outcome.  But,  it  is  true  that  in 
having  been  obtained  from  normal  rathe?  than  t,  tables,  the  values  reported  must 
be  considered  somewhat  inaccurate. 

The  nature  and  extent  of  those  inaccuracies  may  be  judged  readily  cm  the 
basis  of  the  comparison  figures  given  in  table  XXI.  Prom  this  table,  indicating 
the  performance  of  P-131  in  terms  of  both  the  normal  (js)  and  the  t,  distributions, 
it  seems  clear  that  the  inaccuracies  introduced  by  using  a,  arc  modest  and  follow 
a  consistent  pattern.  By  using  &  the  probabilities  of  meeting  the  hypothetical 
criteria  are  overestimated,  but  in  lesser  amounts  as  one  progresses  from  train¬ 
ing,  to  test,  and  then  retraining.  In  the  worst  case  (training)  p  is  enhanced 
only  .020  and  even  is  enhanced  only  .228,  which  is  modest  considering  its 
extreme  sensitivity.  Such  overestimation  with  respect  to  criteria  is  easily 
understood  to  result  from  the  faot  that  at  this  pilot’s  high  level  of  skill  the 
probability  associated  with  a  given  z  is  nearly  always  greater  than  tbs  prob¬ 
ability  associated  with  jfc  of  the  same  value.  Because  fixed  criteria  are  refer¬ 
ences,  tha  z  and  t,  values  in  each  instance  are  the  same. 
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Qa  the  other  hand*  by  using  &  the  probabilities  of  mooting  former  pecc 
levels  ere  underestimated,  apparently  ct^oln  in  lesser  amounts  as  erne  progresses 
f*os  test  to  retraining.  Ones  again  the  differences  are  not  disturbingly  great, 
with  p  depreciated  only  .058  In  the  worst  case  and  the  other  indicants  (p  and  j>) 
relatively  loss  affected,  in  view  of  thair  sensitivity.  This  underestimation 
results  from  the  fact  that  in  computing  the  probabilities  with  respect  to  former 
Pe^=  levels*  the  reference  level  itself  changes  with  the  use  of  rather  than  t., 
so  that  it  is  generally  less  stringent  than  that  obtained  by  the  use  of  jj,  With 
the  unchanged  standard  deviation  estimate  as  the  divisor  the  calculated  value  of 
&  is  necessarily  less— evidently  sufficiently  less  (with  the  frequently  oodsot 
probabilities  found  in  this  case)  to  reduce  the  more  inclusive  probabilities  be¬ 
low  that  obtained  by  using  j;. 

Thus,  in  having  used  tbs  normal  rather  than  the  X  distribution  as  a  model 
criterion-referenced  probabilities  are  slightly  overestimated  and  former  p  qc 
lovel-refereaced  probabilities  are  slightly  underestimated.  However,  as  was*7 
supposad,  the  differences  of  interest  are  not  particularly  affected  by  these 
inaccuracies.  When  calculated  by  t,  the  losses  in  capability  of  P-131  to  coot 
criteria  in  test  compared  with  endrof~t raining  capability  ar@  6.9,  7.8,  and 
100$  for  pj,  f>  and  i>,  respectively.  These  compare  favorably  with  the  parallel 
percentages  of  7.0,  9=3  and  100  given  in  table  IX  for  the  &  analysis.  The 
losses  (by  $)  in  capability  in  test  to  meet  the  former  p^  level  of  17. 9» 

36„7  end  1GC$  axe  also  consistent  with  the  parallel  percentages  of  24.0,  46.3 
and  100$  given  in  table  XI.  The  gains  (by  Jj,)  in  naxinua  retraining  capability 
to  meet  the  criteria  of  2.9,  9*7  and  84.7p>  axe  somewhat  greater  than  those  of 
1*4,  4.6,  and  33.3  reported  in  table  XIX.  Also,  the  losses  (by  X)  in  raadm ra 
retraining  capability  to  natch  former  p  nc  levels  of  4.5*  14.9*  and  72.5  are 
somewhat  less  than  the  parallel  percentages  of  6.1,  20.0,  and  82.9  shown  in 
table  XVIII.  But,  the  primary  effects  rssain  the  same  as  previously  given  in 
all  instances.  Evidently,  on  the  basis  of  the  data  from  P-131,  the  discrepancy 
in  findings  on  retraining  (pc  vs  p  «)  is  not  likely  to  any  significant  eoetont 
the  consequence  of  having  used  the*  normal  rather  than  the  X  distribution  as  a 
model. 

Che  final  concern  regarding  the  computation  of  the  basic  probabilities 
has  to  do  with  the  calculation  of  those  for  the  test  mission.  Obviously  only 
one  data  point  exiota  for  each  parameter  and  to  have  arrived  at  a  probability 
estimate  on  the  basis  of  just  one  measure  cay  seen  to  be  the  apitoco  of  statistical 
bootstrapping— and  perhaps  it  is.  However,  the  fundamental  operation  of  cal¬ 
culating  a  probability  value  to  represent  this  single  measure  in  reference  to 
some  criterion  (given  or  derived  fron  the  individual's  own  performance)  is  no 
different  from  ths  cossaonpl&ce  acceptance  of  the  mm  of  the  raw  scores  as 
providing  the  beat  available  estimate  of  performances  which  would  occur  under 
those  circumstances  in  repeated  testing.  Since  the  mean  is,  in  a  sycnotrioal 
distribution,  also  the  p  cn  value  the  logic  is  precisely  the  case.  It  is  true 
that  the  sampling  distribution  of  p^  is  greater  them  that  for  p#<=n  for  any 
given  a  but  this  is  a  difference  in’a&curaoy,  not  in  kind.  Thus,  *  it  soecs  the 
inherent  nature  of  the  p  05  value  docs  not  introduce  any  difficulty.  What, 
then,  is  ths  difficulty  with  it? 

Instead,  tho  problem  and,  in  turn,  possible  source  of  bias  arises  because 
in  necessarily  Uniting  attention  to  one  neasurssent  (to  avoid  ocntasinatica 
from  learning)  the  opportunity  to  sake  a  direct  estimation  of  variability  is 
sacrificed.  Without  such  an  estimate  the  meaning  (range)  of  the  value 
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(or  csaa)  cannot  ba  end,  tbs  psos  value  or  crli eriojs-ref ez^ie®d  probabil® 
ity  cannot  even  b®  ealeal&tad.  Thaa,  It  la  Important  for  tba  istorpratatlca  of 
the  data  that  eosss  such  estimate  be  available,  Fortunately  In  this  situation, 
as  in  many  others,  a  sufficiency  of  information  m  pesfoxBSsc®  «siste  to  for®  a 
basis  for  estimating  the  variability  shiah  would  bsv©  Men  obasrrod  cm  reposted 
testing*  The  important  question  is  merely,  What  bias  or  other  effects  did  the 
use  of  a  borrowed  variance  estimate  in  calculating  p  introduce?  Or,  pit  tbs 
other  way,  Jdd  the  use  of  a  borrowed  variance  estimate  result  in  an  acceptably 
accurate  es timate  of  tbs  probabilities  in  question? 

In  this  iaatsaee,  rather  than  to  complicate  tbs  analysis  still  further 
by  special  calculations  such  as  night  be  used  in  deriving  the  scat  accurate 
estimates  possible,  the  variances  observed  in  the  last  four  training  trials 
rare  used  in  obtaining  the  test  eiasien  probabilities.  If  tha  test  sissies* 
psrforaance  bad  been  exactly  comparable  to  ead-of-trainizg  performance  it  oould 
be  argued  that  any  errors  arising  from  this  procedure  would  la  readonly  dis¬ 
tributed  sad  so  self-canceling*  But,  tbs  test  udeaien  scores  are,  in  fact,  &m&~ 
what  poorer  than  ^md-of-training  perforates©  scans.  Shies,  together  with  the 
tendency  of  variability  to  chenge  in  the  sms  direction  as  the  moan,  isspliss 
that  the  variances  used  in  calculating  probabilities  for  the  test  mission  perform¬ 
ance  were  more  frequently  imdarostisaticna  of  the  variances  which  would  have  been 
obtained  by  multiple  testing  (if  that  ware  possible)  than  they  wore  cversatimtos* 
Sine®  the  standard  aooras  war®  obtained  by  dividing  the  difforcraos  between  teat 
mission  end  criterion  or  ?9ax  level  refersnoo  values  by  the  setiaated  standard 
deviation  (a),  the  net  result  is  that  there  is  a  tendency  for  probabilities 
derived  froa  scores  batter  than  the  ref@raa.ee  values  to  be  overeat issates  and 
those  from  scores  worse  than  the  ref ©rsnoa  values  to  be  undercut  imtes.  In 
general ,  them,  there  resulted  from  the  use  of  borrowed  variance  a  bias  in  the 
direction  of  overestimating  the  test  mission  capability.  However,  because  the 
degradation  in  performance  «aa  typically  eaall  it  may  b#  cuppoead  that  the  Mtrue" 
varianes  would  not  ba  greatly  different  from  that  used  and,  in  turn,  that  the 
magnitude  of  this  bias  is  slight*  It  is  estisatsd  not  to  have  exceeded  .02 
although  any  such  exact  estimate  in  subject  to  doubt. 

The  aoliftbility  Modal  md  Comdex  IndAosnte.  Finally,  there  &**  sssay 
additional  otters  that  Gould  fee  discussed  concerning  the  way  in  which  overall 
performance  was  indicated.  Of  course,  even  the  use  of  a  profealistic  or  reli¬ 
ability  model  for  arriving  at  overall  indicants  may  ba  considered  dubious  be¬ 
cause  it  is  at  variance  with  psychological  traditione.  However,  assuming  agree¬ 
ment  on  that  general  question,  it  is  at ill  appropriate  to  inquire  whether  the 
particular  way  in  which  performance  was  represented  probabilistically  in  this 
analysis  resulted  in  any  biases  to  be  taken  account  of  in  interpreting  the 
results.  Two  rather  evident  possibilities  of  this  sort  seem  worthy  of  special 
comment. 

The  first  of  these  concerns  ths  attenuation  of  p  with  compensatory  change 
in  the  contributing  elements  and  with  p  unchanged.  This  phenomenon  has  already 
been  described  briefly  in  illustrating  ths  concern  for  task  set  end  the  apparent 
results  of  it  are  exemplified  by  the  relative  gains  and  looses  in  retraining  as 
depicted  by  tables  XIX  and  XX,  But,  for  a  more  complete  understanding  it  asy  be 
helpful  to  consider  this  simplified  eaaaple. 
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Suppose  that  overall  performance  on  a  two-parsmoter  phase  is  of  interest 
and  that  the  probabilities  of  meting  the  forasr  p  qs  level  In  both  par&sators 
sad  on  either  parameter  are  considered  to  satisfy  $Mb  interest.  By  definition 
the  individual  sppro&ehss  the  task  with  an  s^pactation  based  on  past  g-arformaaoe 
of  meting  tbs  form?  level  in  a  passaaier  $5$  of  tho  time  and  mating  it  in  both 
90.25$  of  the  tins  (.950  x  <>950)*  How,  what  assy  actually  happen  to  his  perform¬ 
ance  relative  to  his  foxtser  level?  Clearly,  ha  say  met  it  on  both,  fail  it  on 
both,  or  fail  it  on  one  but  not  on  the  other. 

How,  if  the  estimate  of  the  p  ^5  level,  accepted  as  a  reference,  is  subject 
to  random  variation  then  it  should  m  expected  that  in  the  long  run  successes  (S) 
and  failures  (?)  would  be  equally  distributed  across  ths  two  parameters.  The 
result  would  be  that  the  p  would  always  be  .$50,  as  can  be  illustrated  by  taking 
all  possibilities  with  an  equal  gain  or  loss  of,  say,  .050,  being  assumed.  The 
resulting  p  values  are  1.000,  .950,  .950,  and  .900  for  ths  combinations  SS,  SP, 

PS,  and  PP,  respectively.  The  average  for  the  ps  in  the  total  set,  which  if 
pa  ar®  varying  randomly  must  occur  with  equal  frequency^  is  still  ,950.  (This 
is  true  of  any  collection  of  such  sets  also—i.e.,  for  p. ) 

In  a  similar  manner,  the  p  values  resulting  from  equal  gains  and  losses 
of  .050  say  be  determined  and  contrasted  with  the  expected  value  of  .9025*  Again 
assuming  for  the  sake  of  convenience  equal  gains  and  losses  of  .050  they  are 
1,000,  .900,  *900,  and  .810  for  SS,  SF,  PS,  and  FF,  respectively.  The  average 
for  the  ps  in  ths  total  set  (which  again  if  pa  are  varying  randomly  must  occur 
with  equal  frequency)  is  indeed  .9025,  as  would  be  expected  when  no  change  in 
performance  has  occurred,  But,  note  the  difference  between  this  circumstance 
and  that  of  p.  Whereas  compensatory  gains  and  losses  among  p  values  result  in 
no  deviation  from  th®  expected  p  (the  sum  of  1.000  and  .$00  divided  by  2  still 
equals  .950)  they  de  result  in  a  deviation  fteca  the  expected  p  (.$00  versus  .9025, 
in  this  example).  In  addition,  the  mutual  gain  brings  about  a  change  in  p  which 
is  greater  than  that  brought  about  by  ths  equivalent  mutual  loss.  And,  further¬ 
more,  these  attenuations  of  p  are  compounded  still  more  in  the  calculation  of  p 
for  a  collection  of  such  sets. 

Prom  this  example  it  may  bo  concluded  that  if  any  factor  (such  as  learning) 
enters  into  the  teat  performance  to  disturb  the  random  frequencies  and  magnitudes 
of  gains  and  losses  occurring  in  the  several  parameters  it  will  be  duly  reflected 
in  th©  complex  indicants  p,  p,  and  p.  P  and  p  will  be  shifted  up  or  down 
equally  by  th©  same  amount  of  mutual  gain  or  loss  in  p  and  will  be  unaffected  by 
compensatory  shifts.  However,  p  and  p  will  be  disproportionately  affected  such 
•'■hat  they  will  be  increased  by  mutual  gains  in  p  to  a  greater  extent  than  they 
will  be  degraded  by  comparable  mutual  losses  of  the  same  extent,  and  exactly 
compensatory  gains  and  losses  will  degrade  them.  These  disproportionalities 
vary  directly  with  the  amount  of  shift  starting  withjruite  small  deviations,  as 
in  this  example.  Clearly,  when  compared  with  p  mid  p,  £  and  p  place  a  premium 
upon  the  individual  maintaining  his  relative  performance  on  the  several  measured 
aapeots  of  th©  task  and  reflect  stringently  on  any  tendency  to  gain  in  some  and 
lose  in  other  aspects,  in  tradeoff  fashion. 

How,  having  recognised  the  disproportionalities  which  may  occur  in  p  (and 
p  in  turn)  it  might  easily  be  supposed  that  p  is  not  a  useful  indicant  of  perform¬ 
ance  in  a  complex  (multimsasuro)  task.  However,  that  is  not  th9  contention  hero 
nor  ia  it  a  neoessary  conclusion.  For  example,  it  seems  entirely  reasonable  to 
argue  that  p,  as  a  complex  indicant,  has  exactly  the  properties  that  an  indicant 
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useful  in  reliability  analyses  should  have.  On  the  face  of  it,  at  least,  there 
is  no  doubt  that  it  duly  reflects  the  probability  of  success  in  all  task  aspects*— 
surely  a  useful  concept-— hen  os  ,  that  is  not  the  problem  which  it  occasions.  In¬ 
stead  the  problem  which  it  poses  eeesa  merely  to  be  an  Interpret iva  one  of  relat¬ 
ing  the  possible  variations  in  it  to  traditional  views  on  human  performance. 

Thera  is  a  strong  convention  of  thinking  about  multiple  performances  in  terms  of 
average  values  (arithmetic  means,  etc.)  mad,  of  course,  5  follows  thin  convention. 
Accordingly,  cne  expects  equal  gains  and  losses  to  be  entirely  self-cancelling. 
But,  there  is  no  compelling  reason  why  they  should  be  and,  indeed,  there  may  be 
interpretive  value  in  their  not  being  so.  For,  in  a  broad  view  of  the  human 
performance,  is  not  inconsistency  in  dealing  with  the  task  also  a  reflection  of 
inadequacy  and  unreliability?  If  this  is  so,  then  tha  use  of  p  is  entirely  jus¬ 
tified.  Nevertheless,  its  use  does  require  special  care  in  interpretation,  as 
it  has  been  the  primary  intent  of  the  present  elaboration  to  show.  3 

Thus,  briefly  returning  to  the  analysis  of  retraining  performance,  the 
characteristics  of  p  and  the  insensitivity  of  the  criterion-reference  analysis 
to  certain  losses  taken  in  conjunction  with  the  likely  task  set  of  the  pilots 
(of  0-8  and  0-13)  easily  could  result  in  the  contradictory  findings  obtained. 

As  capable  as  the  pilots  ware  in  meeting  the  criteria  (p  much  greater  than  .959 
in  many  parameters)  the  gains  with  respect  to  parameters  in  which  they  were  least 
capable  would  bring  overall  gains  whereas  the  parallel  losses  in  the  parameters 
of  greatest  capability  would  often  not  be  detected  in  the  analysis  with  respect 
to  criteria.  In  contrast,  the  analysis  with  respect  to  former  level,  being 
sensitive  to  shifts  in  both  directions  and  harshly  reflecting  any  compensatory 
shifts,  would  bring  overall  losses  in  i>  and  the  more  inclusive  indicants  based 
on  it.  Only  p  and  f  would  directly  represent  the  magnitude  of  the  overall  shift 
unaffected  by  the  ray  in  which  the  shift  occurred.  That  did  indicate  some 
loss  suggests  that  gain  in  one  parameter  cost  disproportionately  more  in  another. 

Support  for  this  conjecture  as,  at  least,  partly  explaining  the  contradic¬ 
tory  retraining  results  may  be  derived  from  an  analysis  of  the  relative  order 
of  capability  on  the  several  parameters  of  a  given  phase  in  training  and  in 
retraining.  If  the  supposition  of  a  shift  in  task  goals  by  the  pilots  of  0-8 
and  0-13  is  true,  then  their  retraining  performance  evaluated  by  reference  to 
the  former  p  05  level  ought  to  be  negatively  correlated  with  their  end  of-training 
capability  to  meet  oriteria.  But,  retraining  capability  to  meet  cri'  ,a  con¬ 
trasted  with  end-of -training  capability  would  be  negatively  related  only  if 
shifts  larger  than  interparaoeter  differences  occurred*  otherwise  with  small 
shifts  the  order  of  performance  would  be  maintained  and  the  correlation  would 
be  positive. 


Actually,  the  arithmetic  effects  of  compensatory  shifts  on  those  data  apparently 
are  not  as  great  as  might  bs  supposed.  If  the  p  qc  values  for  P-131  in  his 
best  four-trial  block  in  retraining  are  adjusted  mo  remove  all  effects  of 
compensatory  shift  (by  equalising  parameter  values  within  each  phase  at  the 
level  of  p  for  the  phase)  the  changes  obtained  in  5  and  i>  are  modast.  Thus, 

5  increases  from  .686  to  .730  and  p  increases  from  .068  to  .117.  Furthermore, 
the  losses  in  capability  relative  to  ©nd-of-training  skill  are  only  somewhat 
reduced.  In  $  the  loss  of  .172  ( 20. 0^)„be  comes  .128  (14«9$)  and.  in  $  tha  loss 
of  .329  (82. 9^»)  becomes  .212  (53.4$).  ?  values  are  no*  affected,  of  course. 
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Th®  results  of  just  such  an  analysis,  using  jg.  ve-Iuqs  rather  than  p  for 
greater  discriminability,  is  tabulated  by  pilot  and  retraining  reference  in 
table  mi.  In  all  of  the  33  cases  represented  by  this  table  there  are  only  9 
in  which  the  critical  order  ©?  performance  per  the  levels  (with  P.95/, 95/ 
the  reference)  is  not  negatively  related  to  training  performance  in  ths 
predicted  fashion.  In  view  of  the  insensitivity  of  the  analysis  to  partial 
tradeoffs  (for  example,  compensatory  shifts  in  only  two  of  three  parameters) 
this  proportion  of  variant  oases  doe^  not  sees  large*  Brace,  it  is  concluded 
that  the  presumed  task  sot  effects  did  occur,  although  not  entirely  consistently 
either  among  individuals  or  araong  phases.  Among  the  pilots,  82,  131,  raid  132 
showed  th®  of foot  sore  than  the  others*  among  phases,  the  effect  is  moat  pro¬ 
nounced  in  separation  and  deorbii,  braking  and  hover,  and  docking  and  least 
pronounced  (or  nonexistent)  in  transposition* 

Fartheraore,  if  transposition  is  discounted,  a  loss  in  capability  to  uest 
former  levels  (p.95/,95)  *s  seen  ^  16  of  the  remaining  21  oases  in  which  an 
invars©  relation*  in  order  of  performance  is  noted.  The  overall  average  shift 
for  all  rases  (including  transposition)  in  which  inverse  ordering  occurred 
represents  a  loss,  the  average  shifts  being  -.448,  -.226,  -»255,  ”.234,  +.042, 
and  +.024  for  pilots  of  CM3  rad  0-13,  respectively.  (®i©  alight  gains  of  P-132 
and  P-133  are  attributable  to  the  influence  of  artress  values  at  the  ©ad  of  train¬ 
ing,  which  has  already  bean  rationed.)  The  comparable  average  shifts  for  cases 
in  which  a  positive  relationship  was  found  consistently  represent  less  loss  or 
more  gain,  except  that  for  P-81  whose  retraining  data  are  discounted.  These 
shifts  are  -.806,  .140,  -.086,  .143,  .050,  and  .097  for  0-8  and  0-13  pilots, 
respectively. 

Although  the  specific  gains  rad  losses  in  phases  do  not  correspond  perfectly 
with  direct  rad  inverse  performance  orders,  this  is  to  be  expected.  A  oospensa- 
tcry  shift  would  not  necessarily  imply  loss  (general  gain  in  all  parsmfeers  could 
occur)  just  as  continuing  the  sas®  performance  order  would  not  always  result  in 
gain  (a  general  loss  could  occur)*  Thus,  it  is  not  contsadad  that  all  gains  and 
losses  are  accounted  for  by  task  set  shifts.  Bather  it  is  suggested  that  sot 
shifts  resulted  merely  in  a  trend  toward  degradation  in  phase  performance  eval¬ 
uated  by  reference  to  the  feruar  p.95  level  in  individual  parameters#  This, 
admittedly  weaker,  relationship  does  appear  verified. 

The  other  special  concern  relating  to  the  way  in  which  overall  performance 
is  represented  has  to  do  with  the  possible  relationships  among  the  several 
measured  aspects  of  task  performance.  In  psychological  analysis  of  day-to-day 
behavioral  complexities  it  is  generally  understood  and  expected  that  some  of 
the  measured  or  measurable  aspects  of  the  performance  of  interest  my  be  in  some 
kind  of  dependent  relationship.  This  dependency  may  result  from  e  contingency 
evident  in  the  elements  of  the  task,  or  from  ease  cossson  skill  roquiressrat,  or 
from  some  other  sauro® — asset ly  how  not  necessarily  being  of  initial  significance 
for  th©  measurement  operations*  The  min  problem  is  the  determination  of  what 
the  relationship  among  the  measures  is.  With  this  information  available  deter¬ 
mination  of  the  basis  for  and  detailed  nature  of  the  relationships  s ©y  then  be 
undertaken. 

In  keeping  with  this  accepted  view  on  ©raplax  tasks  it  is  entirely  appro¬ 
priate  to  inquire  about  the  possible  interdependencies  among  task  aspects  in 
the  present  study.  If  there  ware  task  dependencies,  the  next  and  also  appro¬ 
priate  question  to  be  asked  is,  Bear  are  these  dspsadsnoies  reflected  in  the 
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Relationship  Betiraen  Order  of  Paratasier  Porforaacoo  in  'Prattling  and 
Best  Fcjur-Trial  Block  in  Retraining  (Kendall’s  r) 


Pilot 


Reference/ 


Measure 

81 

82 

83 

131 

132 

133 

(jsean) 

THM 

P  c/o 

MB 

1.000 

-.333 

.333 

.333 

1.000 

.461* 

c/95 

MB 

1.000 

-.333 

.333 

.333 

1.000 

.461* 

95/0 

MB 

.333 

-.333 

-.333 

.333 

1.000 

,250* 

95/95 

MB 

.333 

-.333 

-.333 

.333 

-.333 

-.061* 

LOI 

MB 

mm 

4 

4 

4 

4 

4  * 

MB 

*n» 

4 

4 

4- 

4 

4  » 

NB 

- 

- 

- 

4 

-  * 

MB 

■m 

•M 

mm 

4 

4 

-  * 

sm 

-1.CC0 

.333 

1.000 

.333 

“1.000 

-.333 

-.111 

-1.000 

.333 

l.OOn 

.333 

-.333 

-.333 

•  OCX) 

“1.000 

“1.000 

-.333 

.333 

-1.000 

-.333 

-.556 

-1.000 

-1.000 

-.333 

.333 

-.333 

-.333 

-*444 

BH 

.333 

1.000 

.000 

.667 

.661 

.333 

.500 

.333 

1.000 

.000 

1.000 

.661 

.333 

.556 

-.333 

-.333 

“.667 

-.667 

-.661 

.000 

-.445 

-.333 

.000 

-.667 

-1.000 

667 

.000 

-.445 

Xtek 

NB 

.333 

-.333 

.333 

-.333 

-.333 

-.061* 

NB 

1.000 

1.000 

-1.000 

-.333 

-.333 

.061* 

NB 

-1.000 

-1.000 

-.333 

-i.OOO 

-.333 

-.733* 

NB 

“1.000 

1.000 

“1.000 

-1.000 

-.333 

-.467* 

EE 

1.000 

1.000 

.333 

-1.000 

“1.000 

.333 

.111 

1.000 

.333 

.333 

-1.000 

-.333 

.333 

an 

“1.000 

.333 

-.333 

-.333 

-1.000 

-.333 

-.444 

1.000 

-.333 

-.333 

-.333 

-.333 

-.333 

-.111 

All 

P  c/o 

•  111* 

.733 

.133 

.133 

-.261 

.200 

a8o 

o/95 

.111* 

.733 

.400 

“-Q67 

.200 

.240 

95/c 

-.778* 

-.333 

-.533 

-.261 

-.661 

.000 

-.386 

95/95 

-.111* 

“.400 

-.133 

-.461 

-.266 

-.301 

*  baaed  on  reduced  data 
NB  not  don©  by  this  pilot 
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reliability  model  by  mesas  of  which  overall  capability  is  indicated?  How,  it 
is  already  evident  that  the  simple  probability  model  used  in  the  present  analysis 
to  obtain  p  and  p  and  the  more  inclusive  indicants  based  on  them,  frankly  pro¬ 
vides  no  means  of  introducing  task  dependencies*  Each  parsssetor  p-valu©  is  taicsn 
on  its  own  merits  aa  in  no  way  dependent  on  any  other  end  if  all  are  independent, 
as  is  assumed,  p  and  p  are  accurate  within  tho  limits  of  measurement  error*  On 
the  other  hand,  performance  in  certain  parameters  may  be  related  in  actuality, 
with  the  result  that  to  treat  them  as  independent  is  to  introduce  a  bias*  Ksnoa, 
the  question  is,  To  what  extent  '*cro  performances  in  the  several  parameters 
related  and  how,  if  any,  are  the  results  biased  thereby?  Once  again  it  is  not 
possible  to  give  a  complete  and  entirely  satisfactory  answer  and  it  is  necessary, 
as  with  previous  issues,  to  fall  back  upon  analysis  or  personal  judgment  in 
formulating  an  opinion. 

By  the  way  of  exploring  the  seriousness  of  this  issue  an  attempt  was  made 
to  quantify  the  interdependencies  among  task  elements  as  represented  by  the 
performance  record*  Since  there  is  no  reason  to  expect  that  whatever  relation¬ 
ships  existing  would  be  the  same  throughout  the  acquisition  of  skill  in  the 
tasks— in  fact,  it  is  more  reasonable  for  them  to  differ— it  was  necessary  first 
to  choose  the  reference  sample  for  the  analysis.  The  last  four  training  trials 
had  already  been  used  as  the  basic  reference  in  the  analysis  so  it  seemed  quits 
appropriate  that  they  be  used  in  this  analysis  aa  wall.  Presumably,  any  intra¬ 
task  relations  found  existing  in  the  last  four  training  trials  of  0-8  and  0-13, 
in  particular,  would  be  indicative  of  the  kinds  of  relationships  to  be  expected 
in  highly  skilled,  relatively  stable  performance. 

Using,  then,  the  same  four-trial  sets  as  had  previously  been  used  in  com¬ 
puting  endrof-training  probabilities,  the  correlations  in  performance  between , 
each  pair  of  parameters  within  each  phase  and  for  each  pilot  of  0-8  and  0-13  were 
obtained.  Kendal. 's  r  seemed  the  most  appropriate  statistio  for  this  purpose, 
bacauae  in  the  three-variate  (parameter)  oase  a  partial  correlation  can  be  com¬ 
puted,  thus,  excluding  from  the  relationship  of  any  two  in  question  any  effects 
of  their  mutual  relationship  with  the  third  factor.  The  outcome  of  this  analysis 
of  the  seven  phases  entailing  two  or  more  parameters  is  presented  in  table  XXIII* 

The  antries  in  table  XXIII  afford  interesting  possibilities  for  understand¬ 
ing  tho  approaches  which  the  pilots  took  in  performing  the  several  phases.  Presum¬ 
ably,  the  pattern  of  the  interoorrelatiens  for  any  particular  pilot  and  phase 
shows  how  he  organized  the  task  elements,  which  he  was  willing  to  trade  off 
performance  in  for  which,  and  so  forth.  An  excellent  example  (almost  idyllic) 
is  the  performance  of  the  braking  and  hover  sequence  (i.e.,  the  lunar  landing) 
by  P-133,  In  one  way  of  describing  it,  this  pilot  may  be  said  to  have  performed 
tho  phase  as  two  independent  tasks— the  rata  (in  displacement  and  impact)  control 
task  and  the  displacement— fuel  task— trading  off  performance  in  the  elements  of 
each  of  these.  In  contrast,  in  the  same  phase  P-132  appears  to  have  traded  off 
fuel  consumption  with  displacement  rate,  while  holding  those  and  the  other  aspects 
rather  independent  otherwise.  It  might  be  very  profitable  for  understanding 
complex  task  performance  to  obtain  detailed  information  from  the  pilots  as  to 
how  they  approached  the  phases  and  then  to  attempt  to  relate  that  information 
to  these  interoorrelatiens.  However,  although  such  sn  analysis  might  be  very 
valuable,  it  is  not  necessary  to  th®  immediate  need. 
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Corr@latiss.va  of  Psrasatera  within  Phases  on  tbs 
last  Ste.r  draining  Crisis  (Xendall'e  T)f 


Pilot 


Phase/ 

Ccspariccn 

81 

82 

83  _ 

131  . 

132 

133 

111 

(esea) 

TLX 

V  z  P 

,000 

-.667 

-.183 

.183 

.183 

.000 

.081 

war 

IS  x  SB 

.333 

(.250) 

.333 

(.333) 

-.667 

(-.707) 

-.333 

(.000)# 

7*333 

(.000)* 

-.333 

(.490) 

-.167 

(.061) 

IS  x  P 

-.333 

(-.250) 

.000 

(.000) 

.333 

(.447) 

1.000 

(1.000)# 

,*333 

(.000)# 

.913 

(.926) 

.374 

(.354) 

fflx  F 

-.333 

(-.250) 

a  000 
(.000) 

.000 

(.316) 

-.333 

(„occ)* 

-1.000 

(-1.000)# 

-.548 

(-.634) 

-.369 

(-.261) 

LOI 

VxP 

.548 

-.183 

-.183 

.183 

-.707 

-.548 

-.148 

SDO 

V  x  T 

T*333 

(.297) 

.667 

(.633) 

.548 

(.657) 

.183 

(.132) 

.333 

(.000)# 

.913 

(.919) 

.385 

(.341) 

V  x  P 

.730 
(.72 2) 

.667 

(.633) 

.333 

(.527) 

*333 

(.310) 

.333 

(.000)# 

-.183 

(.315) 

•369 

(.418) 

I  x  P 

-.183 

(.093) 

.333 

(-.202) 

-.183 

(-.463) 

.183 

(.132) 

1.000 

(1.000)* 

-.333 

(-.414) 

.136 

(.024) 

BE 

Dx  xa 

-.333 

.333 

.333 

.000 

.183 

.000 

.086 

BxXE 

.183 

.000 

-1.000 

.667 

.548 

.000 

.066 

BiP 

.000 

-.333 

.183 

.333 

-.183 

-1.000 

-.167 

I®  x  3E 

-.913 

.000 

-.333 

-.333 

-.333 

-1.000 

-.485 

m  x  r 

-.667 

-1.000 

-.548 

-.667 

-1.000 

.000 

-.647 

ffix? 

.548 

.000 

-.183 

.000 

.333 

.000 

.116 

Bok 

2  x  3B 

.333 

(.527) 

.333 

(.447) 

—•667 

(-.334) 

-.333 

(-.333) 

.333 

(.447) 

.333 

(.527) 

•055 

(.214) 

p  x  xa 

<-.183 

(-.#3) 

.000 

(.316) 

-.707 

(-.447) 

\.707 

(.447) 

.000 

(.000) 

.000 

(.316) 

-.548 

(-.657) 

-.240 

(-.156) 

VR  x  XR 

.548 

(.657) 

-.667 

(-.707) 

.000 

(.000) 

-.667 

(-.707) 

.183 

(.463) 

.017 

(.026) 

ES 

Dx  A 

.000 

(.316) 

.*333 

(.447) 

.333 

(.250) 

.333 

(.333) 

.333 

(.000)* 

.333 

(.250) 

.278 

(.266) 

Dx  CR 

7*333 

(.447) 

.000 

(-.316) 

-.333 

(-.250) 

.000 

(.000) 

1.000 

(1.000)* 

.333 

(.250) 

.111 

(.189) 

A  x  CR 

.667 

(.707) 

.667 

(.707) 

-.333 

(-.250) 

.000 

(.000) 

.333 

(.000)* 

.333 

(.250) 

.278 

(.236) 

t  values  for  partial  r  arts  given  in  parentheses 

#  value  for  partial  r  is  the  logical  consequence  of  th©  relationship  observed, 

although  not  obtainable  zozeorically  bv  the  usual  rules  of  algebra  (msaerator 
or  denoainator  or  both  equalling  zero) 


2b r,  what  t&  of  primary  concern  for  the  esibodslogieal  issue  her®  under 
fiiaoussion  is  simply  whether  sad  in  what  dograa  performacss  in  mcaeursd  task 
elessai®  within  phases  related®  Sow,  iaksa  all  together  the  correlations 
in  table  SOII  do  not  generally  represent  a  high  dagre®  of  relationship.  There 
are,  of  coasts®,  instances  in  which  r  doe®  reach  *$£>7  and  greater  tut  these  are 
relatively  infrequent  compared  to  the  instances  in  which  it  is  .333  or  less. 
Sbriherssor©,  and  nor®  crucial  still,  is  that  a  clear  organization  of  task  elements, 
as  indicated  by  boss  strong  dependencies  (high  ~ )  sad  so so  independencies  (low  r), 
seldom  emerges,  What  is  mch  sore  frequent  is  a  sot  of  mutually  low  correlations 
representing  rather  weak  relationships  ccong  the  task  elesaenta.  Esther  evidently, 
one  oan  hardly  hope  to  has©  a  useful  correction  of  f  and  p  computations  on  such 
weak  contingencies  m  are  of  tent  ins  2  indicated®  Sms,  any  hopes  for  developing 
more  precis®  expxeaBiam  of  the  phase  performance  indicants  sust  be  restricted 
to  only  those  individuals  and  phases  in  which  moderately  strong  relatione  are  in 
evidence.  (There  are  20  eases  in  the  total  of  40  in  which  the  greatest  correla¬ 
tion  is  .657,  or  sore.)  Bat  ever,  correction  in  many  of  those  would  not  be 
a spoolal ly  useful  because  of  the  considerable  independence  still  involved  in 
the  relationship  of  the  most  strongly  correlated  pair  of  parameters. 

It  is  apparent  also  that  even  less  hope  exists  for  finding  a  suitable 
expression  for  each  phase  which  could  ha  used  for  all  pilots  said  so,  would  offer 
great  computational  convenience.  Casual  cross-comparison  of  row  entries  as  well 
as  ths  magnitudes  of  ths  row  means  in  table  XXIII  shows  that  without  question 
the  pilots  were  not  consistent  in  their  approaches  to  a  given  phase.  The  largest 
mean  is  -.647,  indicating  an  inverse  relation  1st  ween  fuel  and  displacement  rate 
in  braking  and  hover,  and  even  in  this  case  there  is  an  exception  (P-133)*  Even 
among  the  20  cases  yielding  moderately  large  maximal  correlations  there  is 
inconsistency  in  the  patterns  to  be  observed. 

Consequently,  it  was  concluded  that  although  expressions  correcting  th© 
computation  of  g  and  p  for  certain  phases  and  certain  pilots  night  he  feasible 
and  bring  about  improved  accuracy — the  infrequency  with  which  significant 
improvement  could  be  gained  and  ths  complications  involved  do  not  justify  doing 
so.  Instead,  it  seeasd  wiser  to  merely  recognize,  as  has  been  done,  that  p  and 
p  for  certain  phases  said  individuals  are  likely  to  be  somewhat  biased  by  the 
failure  to  take  account  of  th®  contingencies  among  task  elements.  Th®  exact 
character  of  the  bias  can  only  be  discerned  from  examination  of  th®  pattern  of 
correlations  and  ie  unique  to  each  instance.  In  only  a  few  such  cases  is  the 
resulting  error  likely  to  be  of  any  size  numerically.  In  no  instance  is  it 
considered  sufficient  to  influence  comparisons  of  a  given  pilot's  performance  at 
various  times  because  the  bias  in  compared  values  presumably  would  bo  consistent. 
These  statements  have  reference,  of  course,  only  to  0-8  and  0-13.  Bsoauso  0-4 
and  0-9  apparently  did  not  near  their  peak  skill,  it  is  unlikely  their  parforse- 
~nca  refloated  a  stable  organization  and  that  biases  due  to  task  element  depend¬ 
encies  exist  in  it. 

In  summary,  then,  certain  analytic  procedures  used  along  with  certain  data 
collection  procedures  prompt  serious  concern  in  the  interpretation  of  the  data. 
There  are  others  which  might  be  supposed  would  cause  a  problem  but  which  on 
examination  are  considered  to  give  little  difficulty.  Among  those  of  concern, 
the  likely  inaccuracies  in  estimation  of  p  resulting  from  th©  use  of  such  a 
email  sample,  particularly  the  influence  of  occasional  cortresae  values,  necessi¬ 
tates  watchfulness  in  the  interpretation  of  deviant  p  values.  It  is  especially 
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likely  that  the  original  capabilities  of  P-132  and  P-133  are  somewhat  undex- 
estiESted  bscsus®  of  extras®  value  a  in  th©  reference  last  four  trials  of  train- 
ing  and  that  those  extras©  values  influenced  the  estimates  about  their  test 
mission  and  retraining  capabilities  as  veil,  l\*rt  hormone,  in  having  been  obtained 
fro®  the  normal  probability  function  rather  than  Student’s  t,  distribution,  all 
probabilities  of  nesting  criteria  are  generous  whereas  probabilities  of  mooting 
former  p#ec  levels  are  underestimated,  feat  mission  probabilities  also  tend  to 
be  overeat ia&tad  by  the  us®  of  a  K  borrowed*  variance  estimate  in  calculating  them. 
Procedural  details  of  the  test  mission  and  in  retraining  say  have  operated  to 
degrads  performance  in  them  somewhat.  The  disruption  in  plans  almost  certainly 
degraded  P“8l»e  retraining  performance.  On  these  grounds  it  may  be  supposed  that 
with  soma  individual  deviation  the  capabilities  of  0-8  and  C-13  to  meet  criteria 
are  generally  ovarestimatad  in  th©  results  presented,  while  their  capabilities 
in  test  and  retraining  to  meet  former  p  qc  levels  are  generally  underestimated. 
However,  in  considering  p  and  the  more  inclusive  indicants  Involving  it  additional 
account  must  be  taken  of  the  disproportions! it les  inherent  in  as  compared  with 
p,  Kid  the  occasional  modest  inaccuracy  in  estimates  resulting  from  treating  task 
elements  as  independent. 

Although  some  sampling  bias  is  considered  likely  in  retraining,  owing  to 
the  selection  of  the  best  four  trials  for  analysis,  this  effect  is  discounted 
as  roughly  compensated  for  by  diminished  motivation.  The  use  of  dual  criteria 
for  retraining  performance  complicates  interpretation  bit  is  not  a  source  of 
bias.  In  general,  the  data  do  not  suggest  that  any  bias  of  consequence  is  likely 
to  have  resulted  from  asymmetry  in  distribution  of  cases — that  is,  failure  to 
reflect  normality  in  this  way.  Missing  data  is  not  a  problem. 


3&g. 

Having  considered  the  more  specifio  data  collection  and  analytic  procedures 
and  their  probable  impact  on  the  results,  it  remains  to  deal  similarly  with  cer¬ 
tain  more  general  issues  affecting  the  broader  implications  of  the  study.  These 
are  matters  which  strongly  influence  the  rang©  of  applicability,  or  general  inabil¬ 
ity  of  the  findings.  Among  them  are  such  varied  concerns  as  statistical  signif¬ 
icance,  intervening  activities,  skill  differentials,  the  use  of  simulation,  and 
test  personnel  motivation.  Eaoh  of  these  will  be  treated  briefly,  in  turn,  in 
the  following  discussion,  the  primary  intent  being  to  express  the  position  to  bo 
taken  in  the  subsequent  statement  of  findings.  It  is  not  intended  that  the 
comment  on  these  topics  included  here  b®  a  comprehensive  treatment-— for  that  is 
beyond  the  scope  of  this  report.  Furthermore,  it  should  be  recognized  at  the 
outset  that  alternative  views  to  those  presented  are  possible  because  these  are 
debatable  matters. 

Statistical  Significance.  In  the  preceding  brief  description  of  analytic 
methodology  it  has  already  been  noted  that  conventional  tests  of  significance 
for  skill  retention  effects  are  simply  not  feasible  in  view  of  the  structure  of 
the  data.  It  was  not  possible  to  combine  the  performance  of  even  two  pilots  to 
arrive  at  the  necessary  error  estimate  for  suoh  a  test  because  (in  the  language 
of  experimental  design;  all  individuals  represent  unique  treatments.  Further¬ 
more,  although  a  specialized  adaptation  of  the  analysis  of  variance  technique 
might  bo  applied,  the  complexity  of  the  data  structure  makes  doing  so  inadvisable. 
Therefore,  it  seemed  best  to  merely  examine  the  pattern  of  results  obtainod  with 
an  eye  to  the  internal  coherence  of  the  data  and  correspondence  with  reasonable 
expectations  concerning  the  likely  affects  of  known  factors  involved. 
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Sow,  of  course,  there  are  those  who  take  the  position  that  such  a  course 
of  action  is  not  acceptable— that  not  to  determine  levels  of  statistical  sig¬ 
nificance  for  differences  observed  is  tantamount  to  unscientific*  After  all, 
if  one  does  not  know  the  level  of  significance  attaching  to  an  observed  differ¬ 
ence  how  oan  the  meaning  of  that  difference  ho  known?  There  is  no  fundamental 
quarrel  with  this  position-— 00  far  as  it  goes*  Most  certainly,  a  real  advantage 
in  precis®  interpretation  and  enhanced  meaningftilness  does  accrue  from  tbs 
expression  of  statistical  significance*  However,  the  difficulty  with  this  simple 
view  of  data  interpretation  is  that  it  does  not  go  far  enough  to  comprehend  the 
realities  of  observational  complexity  in  many  situations  of  interest.  Also, 
it  does  not  allow  for  the  investment  of  meaningfuiaess  by  other  methods,  and 
presumes  that  only  the  statement  of  statistical  significance  can  serve  that 
purpose.  And  yet,  oftentimes  it  is  quite  important  to  evaluate  in  sos®  way  the 
regularities  of  complex  data.  Should  all  hope  of  doing  so  be  given  up  for  the 
lack  of  a  ready  modal  for  determining  the  likelihood  that  a  particular  effect 
occurred  by  chance? 

In  a  broader  view  of  the  data  interpretation  problem  the  moat  appropriate 
answer  mat  surely  be,  No!  Although  t-teate,  F-ratioa  and  tha  like  have  great 
utility  when  they  oan  be  applied,  to  argue  that  without  them  nothing  can  bs  don® 
is,  frankly,  to  denounce  a  great  variety  of  data  as  of  no  use.  It  is  to  sacri¬ 
fice  a  host  of  potentially  useful  observations  on  the  altar  of  significance  in 
an  unthinking  fervor  of  statistical  reverence.  Instead,  it  seems  that,  with  all 
due  regard  to  the  difficulties  involved,  the  effort  can  and  should  be  made  in  any 
case  to  examine  the  organisation  of  the  results  obtained.  For,  a  degree  of 
meaning  also  may  be  attached  when  a  set  of  interrelated  observations  also  shows 
a  high  degree  of  internal  coherence  or  an  expected  pattern  of  magnitudes.  What, 
after  all  does  or  should  an  investigator  do  on  encountering  a  collection  of  t.  or 
F  values  not  quite  significant,  but  all  readily  interpretable  on  extra-statistical 
considerations?  Should  he  conclude  that  no  meaning  is  to  be  derived  from  the 
results?  To  do  so  seems  not  only  unwise,  but  wasteful. 

Accordingly,  the  position  adopted  in  this  report  is  that  if  a  collection 
of  observations  follow  rather  consistently  reasonable  expectations  concerning 
their  order  of  magnitude,  it  should  be  concluded  that  the  existence  of  the  effect 
is  thereby  demonstrated.  However,  because  the  degree  of  consistency  required 
for  acceptance  of  a  particular  conclusion  is  a  matter  of  judgment,  a  sufficiency 
of  information  is  always  presented  to  enable  the  reader  to  himself  examine  the 
consistency  involved  and,  thus,  to  evaluate  the  merits  of  the  conclusion  given. 

.  On  the  other  hand,  the  adoption  of  this  policy  should  not  be  taken  to  imply 
that  tests  of  significance  on  these  performance  data  are  believed  to  bo  impossible. 
Even  though  the  data  are  presented  in  an  unusual  probabilistic  fashion  it  is  still 
rather  obviously  possible  to  apply  counting  or  frequency  methods  to  them  to  arrive 
at  some  rough,  indication  of  significance  and,  in  turn,  broad  generalities.  For 
example,  one  oan  readily  calculate  the  probability  of,  say,  six  out  of  six  crew¬ 
members  not  reaching  the  same  p0  in  test  as  in  training*  Such  calculations  were 
not  considered  necessary  for  interpretation,  but  some  readers  may  wish  to  do  so 
for  their  own  interest.  Finally,  beyond  these  offhand  methods,  more  refined 
techniques  for  testing  the  significance  of  differences  in  the  same  Individual's 
performance — say,  between  training,  tost,  and  retraining— also  are  possible. 

But,  because  further  development  and  study  of  them  before  actual  application  is 
considered  desirable,  no  attempt  was  made  to  use  them  in  this  analysis. 
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Intervening  /otiviiiea.  Turning  attention  new  to  broad  issues  oosoamM 
with  data  collection  procedures,  a  so  far  imdiscusaad  concern  of  great  possible 
significance  ie  that  of  the  activities  intarvening  training  end  skill 

retention  test.  In  early  consideration  of  memory  sad  forgetting  psychologists 
inclined  to  the  supposition  that  the  mere  passage  of  ties  was  sufficient  to 
acccpjnt  for  the  degradation  in  skill  or  recall  noted  subsespjently.  However, 
recent  studies  in  which  the  nature  of  the  activities  engaged  in  during  the 
retention  period  have  bean  varied  have  cast  serious  doubt  upon  this  position* 

Thus,  today  it  is  ©oassonly  held  that  these  intervening  activities  have  much 
more  influence  on  subsequent  performance  than  the  length  of  time  per  so. 

Mindful  of  the  importance,  then,  of  intervening  activities  for  a  close 
interpretation  of  the  data,  upon  reporting  for  the  skill  retention  test  each 
pilot  was  asked  informally  to  describe  in  a  general  way  what  he  had  bean  doing 
since  the  completion  of  training,  Eo  attempt  was  mad®  to  estimate  specific 
numbers  of  hours  involved  or  to  go  into  sensitive  details.  Each  was  aatood  also 
to  what  extent  he  thought  thesa  activities  wore  helpful  in  maintaining  his 
capability  to  perform  the  lwiar  lending  mission. 

In  brief,  the  consensus  of  their  responses  was  that  each  had  engaged 
throughout  the  period  in  normal  test  pilot  activities,  including  conferences, 
analysis  of  engineering  details,  test  planning,  end  flight  in  high  performance 
jet  aircraft.  Furthermore,  they  felt  that,  in  particular,  their  periodic 
flights  in  high  performance  aircraft  brought  substantial  benefit  in  maintaining 
their  capability  to  deal  with  a  complexity  of  factors  under  time  pressure,  as 
was  required  in  certain  portions  of  the  test  mission.  They  did  not  feel  that 
this  benefit  accrued  from  specific  similarities  in  task  requirements  but  rather 
from  the  maintenance  of  more  general  habits  an  might  ba  characterized  as  alert" 
ness  to  requirements  for  time-sharing  and  rapid  shifts  of  attention,  and  for 
rapid  response.  Several,  when  later  asked  specifically,  did  not  feel  that  any 
particular  dissimilarities  in  required  tasks  had  evident  negative  effects  on 
their  performance  (they  are  used  to  dealing  with  such  dissimilarities)  and  con¬ 
firmed  their  earlier  convictions. 

If  the  pilots*  views  of  the  effects  of  the  intervening  activities  are 
taken  at  face  value  (and  there  is  no  evident  reason  they  should  not  ba)  than 
what  implications  do  they  have  for  the  study  findings?  Surely  any  statement 
regarding  the  degree  of  skill  loss  encountered  or  to  be  expoeted  must  ba  qualified 
by  reference  to  the  presumed  benefits  of  these  intervening  tasks.  This  implies 
that,  other  things  being  the  same,  pilots  whose  intervening  dirties  do  not  involve 
tasks  offering  similar  beneficial  effects  can  be  expected  to  perform  less  well. 

It  also  follows  that  the  present  estimate  of  skill  retention  is  a  generous  one 
compared  with  what  might  be  derived  from  other  circumstances  involving  different 
intervening  activities.  On  the  other  hand,  that  is  not  to  state  that  even  greater 
skill  retention  could  not  be  obtained  with  some  still  more  beneficial  intervening 
task  requirements. 

Skill  Different ials.  Still  another  general  feature  of  study  procedure 
having  great  potential  bearing  on  the  interpretation  of  the  results  is  the  level 
of  skill  attained  prior  to  test.  This  factor  has,  of  course,  already  been  con¬ 
sidered  in  the  presentation  of  training  performance  and  the  necessity  of  distin¬ 
guishing  0-4  and  0-9  performance  from  0-8  and  0-13  performance  in  this  respect 
soons  established.  Furthermore,  it  now  appears  that  even  the  best  trained  crews, 
in  general,  had  not  quite  reached  the  limit  of  their  capability.  However,  beyond 


86 


the  gross  differences  la  crew  capcbilitias  is  tbs  tm  at phases  there  ere 
also  tka  sor®  subtle  diffsrs&css  la  oapsMlitiss  of  areas  end  individuals  within 
study  pk&s©»„  ©is  partiasai  concern  hsrs  ie  Khst  da  thas©  differences  i^ply  t 
porfeicularly  in  tbs  case  of  0-8  sad  0-13  of  prisary  interest,  far  tbs  eoaslc®- 
less  to  ba  drsm. 

Jo  brief,  it  is  contended  that  tbs  dlffsrtssifflsa  in  individual  pilot  capa¬ 
bility  to  parlors  the  wari ma  oisalea  tasks  as  mil  as  tbs  differs©  soa  among 
pilots  and  arms  in  capability  to  perform  4b®a  greatly  reduced  (or  allainatsd) 
ths  possibility  of  shewing  the  exact  effects  of  both  retention  interval  end 
testing  ordar.  Th©  pilot®  of  0-8  and  0-13  bad  indssd  reached  similar  lovels  of 
skill  as  ehosa  by  tbs  overall  indicants  but  thsy  mre  still  not  the  oaaa  sad 
they  differed  oonRidcrably  on  the  tasks  of  relative  expertise  and  deficiency. 

Aa  a  result  the  level  of  original  skill — tocnas  to  ba  a  pogorful  dsrfeaminant  of 
rotanticar— east  be  assussd  tc  have  opsr&tad  variously  to  eitaar  oihance  or  degrade 
the  relative  performance  of  tha  individual  a  and  cram.  Since  level  of  learning 
effects  are  relatively  great  it  cay  ba  supposed  that  they  ovorahadowsd  suob 
lessor  affects  as  eight  otherwise  have  been  observed.  Thus,  for  oaasple,  it  my 
mil  be  that  differences  in  tha  losses  of  P-8l  end  P-131  on  test  sight  have  been 
noted  had  those  pilots  not  differed  in  a  fashion  parallal  to  in©  hypothetical 
effects  of  retention  interval.  If  P-131  had  been  exactly  as  ©killed  on  P-8l  ha 
night  have  eaporienosd  even  greater  loss  than  ha  did.^ 

That  it  ms  possible,  even  though  skill  differentials  did  exist,  to  distin¬ 
guish  the  apparent  effect  of  testing  order  such  that  the  first-tested  did  leas 
well,  is  a  testimony  to  the  relatively  strong  beneficial  effects  of  opportunity 
to  observe  and  participate  in  a  prior  performance .  Bat,  that  retention  interval 
had  no  clearly  distinguishable  affects  implies  only  that  it  did  not  have  as  strong 
an  Influence  aa  order  of  test  or  original  learning  level’ — not  that  it  had  no 
effect.  Thus,  it  must  ba  understood  that  because  of  th©  lack  of  exact  control 
over  skill  level  no  conclusion  camcernlng  retention  interval  beyond  it®  ral&tiv© 
strength  as  a  variable  can  be  assorted. 

Simulation  Versus  Operations.  One  final  issue  relating  broadly  to  the 
fundamental  nature  of  the  study  is  the  question  of  simulation  versus  real  opera¬ 
tions.  This  matter  is  popularly  considered  of  such  significance  that  for  cany 
readers  the  viewpoint  taken  on  it  will  primarily  determine  tha  credibility  of 
anything  else  which  may  be  asserted  about  the  study.  And,  of  course,  general 
opinions  aa  the  appropriateness  of  using  simulation  techniques  in  aerospace 
system  research  and  design  range  all  the  way  from  near  rejection  and  grudging 
willingness  to  see  them  used  in  very  prelinin&ry  dsvalcpaw.t  work  to  enthusiastic 
acceptance  of  them  aa  of  great  value  throughout  all  phases  of  research  and 
development,  including  the  final  determination  of  operational  readiness. 


Somewhat  confusingly,  and  at  variance  with  conventional  expectations,  at  high 
reliability  levels  a  greater  degree  of  skill  may  be  associated  with  greater 
(not  leaser)  looseo  In  reliability.  The  man  who  is  cost  reliable  has,  in  a 
sense,  moat  to  lose. 
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This  study  is,  frankly,  profaned  upon  the  bslisf  that  e  insulation  t@ohftiq.ues 
afford  a  useful  means  of  gaining  data  on  and  indioat  ions  of  human  performance 
capability  otherwise  not  obtainable  or  obtainable  only  at  great  cost  and  effort. 

It  is  not  the  intent  of  these  remarks  to  attempt  convincing  those  who  lack  this 
degree  of  acceptance  of  simulation  methodology  that  they  should  adopt  a  more 
positive  view.  But,  in  order  to  be  in  the  best  possible  position  to  realistically 
interpret  the  results  obtained  it  is  necessary  to  consider  the  probable  conse¬ 
quences  of  the  simulation  methodology  employed.  For,  to  accept  results  obtained 
by  -sans  of  simulation  without  regard  to  the  limitations  inherent  in  the  method 
would  constitute  an  indefensible,  blind  acceptance.  In  fact,  it  is  only  whan 
the  results  obtained  by  simulation  are  interpreted  with  o are  to  insure  that 
account  is  taken  of  the  influences  of  the  method  on  the  data  that  the  utility 
of  the  method  can  b®  realised, 

What,  then,  are  the  probable  affects  on  the  data  deriving  from  their  being 
taken  from  a  aisailated  performance  rather  than  a  real  operation?  Or,  stated 
another  way,  how  would  pilot  performance  have  differed  if  the  test  mission  had 
been  a  real  lunar  landing  mission  and  the  retraining  performances  had  been 
imbedded  in  real  lunar  missions?  For  that  natter,  it  may  be  questioned,  as  Trail, 
how  the  estimate  of  end-of-training  capability  would  have  changed  if  real  missions 
had  been  used  instead. 

In  formulating  and  considering  answers  to  these  questions  it  is  convenient 
to  distinguish  three  main  kinds  of  ways  in  which  performance  could  be  influenced. 

In  the  first  pl^oe,  specif io  task  features  may  differ  between  the  simulated  and 
real  situations,  A  simulated  situation  designed  to  replicate  as  exactly  as 
possible  the  real  situation  (as  is  the  traditional  approach  in  the  design  of 
training  simulators)  might  be  expected  to  produce  the  same  results  as  the  real 
situation.  However,  it  is  the  fact  that  simulators  never  quite  yet  reproduce 
the  real  environment*  typically  ihere  are  still  missing  from  them  complexities 
of  the  visual  world  and  of  the  motion  dynamics,  if  not  some  of  the  simpler 
operational  aspects.  Consequently,  it  may  be  supposed  that  as  a  result  of  suoh 
speoific  dissimilarities  performance  will  be  modified — either  enhanced  or  degraded, 
depending  on  the  nature  of  tbs  differences. 

In  this  study,  although  the  simulation  is  considered  to  have  been  well  done 
and  as  nearly  complata  in  task  specifics  as  current  technology  permits,  it 
nevertheless  departed  to  some  degree  from  reality,  particularly  with  respect  to 
motion  dynamics  and  visual  cues  in  the  lunar  landing.  Furthermore,  many  of  the 
environmental  factors  of  apace  flight,  suoh  as  weightlessness  and  radiation, 
wore  not  included  and  realistic  attention  tc  life  support  tasks  was  not  required. 
Finally,  perhaps  most  important  of  all,  while  tho  specific  task  details  were 
chosen  to  duplicate  in  a  general  fashion  early  planning  for  the  real  Apollo  even 
these,  now  somewhat  obsolete  task  details,  were  modified  deliberately  in  the 
direction  of  greater  manual  oontroi. 

The  net  effect  of  all  these  specific  differences  is,  of  course,  a  matter 
of  conjecture  but  some  reasonable  implications  can  be  drawn.  The  most  significant 
is  probably  that  the  pei*fonaanos  observed  cannot  bo  taken  as  exactly  Indicative 
of  that  to  be  obtained  in  any  particular  real  system,  including  the  Apollo. 

Bather,  to  the  extent  otherwise  valid  the  obtained  reliabilities  may  be  taken 
only  as  broadly  indicative  of  the  orders  of  magnitude  which  may  be  expected  in 
a  mission  of  the  nature  used.  Ho  precise  generalisation  to  a  particular  system 
may  be  based  on  these  data  unless  it  is  established  on  dose  analysis  that  the 
details  of  the  tasks  of  interest  were  exactly  oonparable. 
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Furthermore,  although  the  absence  of  environmental  stressors  and  life 
support  requirements  reasonably  may  be  expected  not  to  have  biased  the  perform¬ 
ance  from  that  of  normal  operations,  there  is  always  the  possibility  in  real 
operations  of  a  problem  in  these  areas.  Such  an  occasional  problem  could  result 
in  a  degradation  in  other  aspects  of  performance.  That  happenstances  of  this 
nature  wore  missing  from  this  simulation,  suggests  that  the  obtained  performance 
estimates  are  somewhat  generous,  but  in  an  unpredictable  fashion  as  to  what  por¬ 
tions  of  the  mission  might  bs  involved.  On  the  other  hand,  the  speoifio  defi¬ 
ciencies  in  simulation  detail,  particularly  for  0-4  and  0-9,  probably  operated 
to  make  the  braking  ani  hover  phase,  and  perhaps  docking,  unrealistically  diffi¬ 
cult' — certainly  this  is  the  view  expressed  ty  the  pilots. 

Then,  in  addition  to  the  speoifio  differences  existing  between  simulated 
and  operational  situations,  there  are  also  nonspecific  differences  arising  from 
the  performance  contexts.  Two  kinds  of  contextual  factors  may  be  usefully 
identified  and  separately  considered. 

The  first  of  these  is  the  possible  effeot  of  disturbances  in  performance 
continuity  which  often  o^mr  in  a  simulated  operation.  Unlike  the  real  mission 
which  unfolds  in  continuous  fasiion,  each  particular  phase  following  in  an  orderly 
way  upon  the  previous,  the  simulated  mission  is  often  characterized  by  unrealis¬ 
tically  abrupt  changes  in  task  requirements,  by  the  omission  of  less- impart  ant 
or  nonessential  tasks,  by  the  elimination  of  longer  time  periods  in  whioh  spec¬ 
ified  mission  activities  are  not  required,  and  even  sometimes  a  juxtaposition 
of  task  requirements.  All  of  these  typical  departures  from  realism  were  involved 
in  this  study  and  so  it  is  essential  to  inquire  hot?  they  may  have  affected  the 
results. 

In  general,  with  respect  to  the  test  mission,  it  is  supposed  that  the  net 
affects  of  these  deviations  from  realism  were  to  somewhat  degrade  performance. 

All  of  them  may  be  described  in  psychological  terms  as  disturbances  which  tend 
to  break  or  disturb  the  development  of  the  pilot’s  set  for  performing  a  given 
task  by  prompting  his  precipitous  attention  to  the  new  activity  without  the 
bozisfit  of  "tlio  usual  Ion  ’tiiss  End  parlor  cue  lug  ovon/ts  yhioh  I19  sesjt  h?.7s 

learned  to  depend  upon.  In  contrast,  training  and  retraining  performance  would 
perhaps  be  much  less  disturbed  in  this  way — at  least  that  following  the  initial 
performance  of  a  given  phase.  Indeed,  the  opportunity  to  repeat  a  performance 
immediately  may  have  resulted  in  an  enhancement  due  to  the  maintenance  of  set 
and  better  immediate  recall  of  task  details  (especially  in  more  difficult  phases) 
up  to  the  point  of  fatigue  and  boredom.  That  some  boredom  probably  occurred 
in  the  last  portion  of  retraining  has  already  been  mentioned. 

The  second  kind  of  nonspecific  difference  resulting  from  context  concerns 
the  emotional  and  motivational  concomitants  of  performance.  It  is  just  plainly 
and  irrevocably  true  that  simulated  operations  (including  this  ons)  generally 
do  not  involve  the  pilot  in  the  same  degree  of  risk  to  personal  safety  and 
potential  loss  from  failure  as  do  operational  missions.  Furthormore,  on  th® 
realization  of  their  simulated  nature,  exactly  the  same  degree  of  importance  is 
seldom  attached  to  success  in  simulated  missions,  and  even  if  it  is  the  reasons 
are  different.  Consequently,  in  most  oases  the  emotional  responses  as  well  as 
th©  motivations  of  the  individual  performing  a  simulated  mission  are  different 
from  those  of  an  individual  performing  a  real  mission. 
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Now,  on  the  strength  of  those  differences  in  emotion®  and  isotivation  it  is 
often  supposed,  i  ha  re  fore,  that  the  individual  will  perforce  the  simulated  mission 
soamhat  leas  adequately  than  th  real  mission.  This  follows  from  the  002350a 
supposition  that  greater  personal  concern  results  in  superior  performance.  Bor- 
sver,  the  difficulty  with  this  simple  view  is  that  considerable  experimental 
evidence  shows  that  there  is  not  a  direct  relationship  between  performance  ade¬ 
quacy  Mid  personal  concern.  Instead,  it  appears  that  increasing  personal  concern 
brings  perfr  ^nce  ix^rovmsnt  at  a  diminishing  rate  up  to  a  certain  point  beyond 
which  it  brings  performance  degradation  at  an  increasing  rat  There  seems  little 
doubt  that  increasing  motivation  and  emotional  responses  t'  ask  beyond  a  cer¬ 
tain  point  is  not  only  of  no  value,  but  may  ba  detrimental,  _.»t,  within  a  consider¬ 
able  range  about  the  point  of  opiiraxa,  changes  in  personal  Involvement  result  in 
only  slight  changes  in  performance.  Thus,  In  order  to  estimate  the  probable  rel¬ 
ative  consequences  for  performance  of  motivation  and  emotionality  in  the  simulated 
mission,  contrasted  with  a  real  mission,  it  is  necessary  to  judge  where,  with 
respect  to  this  optimum  tangs,  the  respective  degrees  of  such  involvement  are 
place d—not  simply  that  they  differ. 

Previous  experience  with  space  operations  suggests  that  the  personnel  be¬ 
come  keenly  and  personally  involved  and  experience  some  anxiety,  bat  that  these 
factors  do  not  teach  such  proportions  as  to  much  degrade  performance.  This 
suggests  that  space  missions  are  typioally  performed  within  the  optimum  range  of 
personal  involvement,  as  is  to  be  expected  because  considerable  development  effort 
and  operational  planning  is  directed  toward  this  goal.  What,  then,  can  be  said 
of  the  involvement  of  the  test  personnel  in  the  present  simulation  study? 

Again,  aa  a  matter  of  judgment  Mid  estimation,  it  seems  likely  that  the 
pilots'  motivation  for  excellent  performance  and  their  anxiety  in  the  retention 
test  mission  were  such  as  to  place  them  within  the  optimum  range.  Because  of 
the  inherent  challenge  to  their  professional  skills  the  test  afforded  and  the 
considerable  attention  given  it  on  the  part  of  certain  professional  associates 
the  pilots  wore  quite  eager  to  perform  well.  They  viewed  the  test  as  having  at 
least  seas  implications  for  their  career.  Accordingly,  so  far  as  motivational 
and  emotional  concomitants  are  concerned  thtlr  performance  in  the  skill  retention 
test  mission  is  considered  comparable  to  that  of  an  operational  mission. 

On  the  other  hand,  the  same  cannot  be  asserted  about  their  retraining  and, 
perhaps,  about  their  ondr-of-t raining  performances.  While,  they  no  doubt  began 
retraining  with  considerable  interest  in  achieving  their  best  performance  the 
situation  was  not  the  same — they  had,  in  a  sense,  already  taken  and  passed  the 
test  and  the  .  cBt  was  anticlimax.  Hence,  it  is  supposed  that  while  their  personal 
involvement  probably  continued  at  nearly  the  same  representative  effectiveness 
level  for  the  first  block  or  two  of  trials  on  the  first  day  of  retraining,  there¬ 
after  it  diminished  to  the  point  of  noticeable  effects.  On  similar  grounds  it 
scene  doubtful  whether  the  pilots  performed  the  very  last  portion  of  training 
(having  long  before  demonstrated  regular  capability  to  meet  the  prescribed 
ncminals)  with  the  optimum  and  representative  degree  of  involvement.  Thus,  it  is 
concluded  that  although  motivational  and  emotional  faotors  attributable  to  the 
simulated  performance  situation  probably  did  not  bias  the  skill  retention  tost 
they  probably  did  result  in  some  underestimation  of  capability  at  the  end  of 
training  and  after  extensive  retraining. 
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In  summary,  "^ith  respect  to  the  problem  of  general inability,  the  position 
adopted  for  the  purpose  of  formulating  conclusions  is  simply  this*  The  lack  of 
traditional  tests  of  significance  docs  not  preclude  the  drawing  of  useful  con¬ 
clusions  providing  this  is  done  with  care  and  the  reader  can  evaluate  the  basic 
for  the  conclusions.  The  activities  of  the  pilots  intervening  between  training 
and  test  had  generalised  beneficial  effects  on  their  test  performance  which  could, 
hue  might  not  be,  duplicated  in  other  skill  retention  circumstances.  The  lack  of 
exact  control  of  level  of  original  skill  rendered  the  study  incapable  of  showing 
the  precise  effects  of  length  of  retention  interval  but  did  not  overshadow  the 
gross  effects  of  retention  per  se  and  individual  testing  order.  Art  finally,  the 
use  of  a  simulated  situation  probably  tended  to  reduoe  the  level  oz  performance 
in  training  and  in  retraining,  as  compared  with  a  real  mission,  but  did  not  much 
effect  test  mission  performance. 


The  Considered  Finditt  > , 

The  foregoing  discussion  of  factors  which  should  be  considered  in  inter¬ 
preting  the  results  obtained  is  not  nacessa.  ily  exhaustive.  Nevertheless,  it 
does  make  evident  tha-1-  •‘•here  r^r®  a  number  of  factors  operating  which  in  some 
degree  did  or  may  have  influenced  either  the  performance  itself  or  the  analytic 
results  and,  therefore,  led  to  either  over-  or  underestimation  of  pilot  capability 
on  occasion.  Some  of  the  factors  identified  are  of  such  a  nature  as  to  bring 
about  fluctuations  by  parameter  and  phase  in  either  direction  in  a  way  difficult 
to  guess  upon  in  any  particular  instance.  But,  since  no  clear  basis  was  found 
for  supposing  them  to  result  in  a  definite  bias  one  way  or  the  other  it  is  reason¬ 
able  to  discount  them  from  further  concern  in  sn  overall  view  of  the  results. 

They  may  well  have  affected  certain  p,  p,  and  p  values,  but  they  probably  had 
littlb  affect  upon  f,  $,  etc. 

On  the  other  hand,  there  were  identified  a  few  other  factors  which  probably 
did  introduce  a  bias  (though  not  necessarily  a  consistent  one)  rather  than  merely 
fluctuations  in  p  values.  These,  naturally,  ir. -luenced  the  overall  indicants  in 
a  parallel  fashion  as  wall  and,  therefore,  they  must  be  considered  in  interpreting 
the  results.  Thus,  omission  of  certain  requirements  (unpracticed  emergencies, 
management  of  life  support  equipment,  etc.)  iendea  to  result  in  performance  over- 
estimation  throughout  the  study.  In  addition,  the  use  of  the  normal,  rathor  than 
the  t,  distribution,  also  brought  an  overestimation  of  capability  to  meet  criteria 
ana  an  underestimation  of  capability  to  meet  former  p^c  performance  levels.  The 
relative  lack  of  precise  goals  and  precise  feedback  in  the  procedures  used  with 
0-4  and  0-9  consistently  placed  these  crews  at  a  disadvantage  compared  with  0-8 
and  0-13. 

Still  other  biases  were  identified  which  influenced  the  results  for  only 
one  or  two  but  not  all  three  main  portions  (trai:  tug,  test,  and  retraining)  of 
the  study.  In  training  the  lack  of  motivational  and  emot  Lonal  co’-soraitants  of 
a  real  mission  operated  to  generally  depreciate  eF'iraated  capability.  In  the 
test  mission  the  break  in  r>st  (resulting  from  time  compression)  and  perhaps  some 
other  procedures  tended  to  depreciate  the  attained  reliabilities,  whereas  the  use 
of  variance  estimates  based  on  training  data  tended  to  elevate  them.  In  the 
retraining,  again  the  lack  of  realistic  motivation  and  emotional  responses  and 
perhaps  certain  procedures  probably  degraded  performance,  but  these  effects  are 
at  least  partially  compensated  for  by  making  the  calculations  with  reference  to 
the  best  four  trials. 
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'The  results  for  certain  pilots,  in  particular,  also  probably  are  biased. 
Because  of  the  extreme  values  recorded  at  the  end  of  training,  it  is  quite  likely 
the  estimated  capabilities  of  P-132  and  P-133  to  meat  criteria  in  training  and  in 
the  test  mission  are  spuriously  low  and  their  actual  gains  (if  any)  in  retraining 
over  training  less  than  indicated.  The  retraining  performance  of  P-8l  was  almost 
certainly  adversely  affected  by  the  other  requirements  imposed  on  him. 

Evidently,  soms  of  the  biases  operated  to  underestimate  capability  while 
others  operated  to  overestimate  it  and  so,  to  some  extent  at  least,  they  tended 
to  cancel  each  other  out.  To  what  extent  they  did  cancel  out  depends  upon  the 
particular  combination  of  amounts  and  directions  of  bias  in  particular  portions 
of  the  study  and  in  the  two  types  of  analysis  (criterion  and  5.95  level).  In 
general,  after  careful  weighting  of  positive  and  negative  influences  according 
to  their  estimated  magnitudes,  it  was  conoludad  that  the  overall  estimates  given 
(i.e.,  p,  p,  f>)  of  capability  of  0-8  and  0-13  to  meet  criteria  in  original  train¬ 
ing  and  in  the  test  mission  are  somewhat  overestimated;  while  capability  to  meet 
criteria  in  retraining  is  slightly  underestimated.  (The  values  for  P-132  and 
P-133  in  training  arid  test  and  for  P-8l  in  retraining  were  discounted  from  this 
generality  as  spurious. )  These  net  effects  are  judged  to  be  quite  small  relative 
to  the  reliabilities  reported  and  not  to  exceed  about  .01  in  J,  In  particular, 
the  probabilities  reported  for  training  performance  are  considered  to  be  very 
close  to  those  which  might  bs  obtained  in  a  real  mission.  Presumably,  the  values 
for  0-4  and  0-9  in  training,  test,  and  retraining  are  similarly  affected.  However, 
in  having  had  less  effective  task  goals  and  performance  feedback  they  probably 
performed  less  well  than  they  might  have  throughout  the  study.  This  is  a  bias 
only  insofar  as  their  performance  is  compared  with  that  of  0-8  and  0-13. 

In  a  similar  fashion  it  was  concluded  that  the  overall  probabilities  of 
0-8  and  0-13  pilots  realizing  their  individual  P.95  levels  in  the  several  aspects 
of  flight  control  in  the  test  mission  and  in  retraining  are  underestimated.  In 
general,  the  amount  of  underestimation  or  net  bias  in  fi  in  these  instances  was 
judged  to  approximate  .03.  The  parallel  overall  probabilities  of  0-4  and  0-9 
pilots  my  have  been  similarly  affeoted,  although  possibly  to  a  lesser  extent 
because  of  their  lesser  original  skill. 

On  the  basis  of  this  evaluation  of  the  influence  of  the  various  interpretive 
factors  it  seems  unlikely  that  any  inaccuracies  in  the  results  are  of  sufficient 
magnitude  to  invalidate  the  main  comparisons  of  interest,.  For  example,  if  the 
test  capability  of  0-8  and  0-13  is  overestimated  more  than  original  capability 
then  the  reality  of  the  observed  decrement  is  more,  not  less,  assured.  Similarly, 
the  pattern  of  supposed  inaccuracies  suggests  that  by  additional  performance  in  a 
real  situation  these  crews  would  have  recovered  their  original  capabilities  to 
meet  criteria  even  more  quickly  than  they  seemed  to  in  this  study.  Furthermore, 
all  of  the  inaccuracies  described  are  sufficiently  small  as  to  permit  a  rather 
precise  impression  of  the  aotual  reliahilit.  a  to  he  expected  in  a  real  mission. 

It  is  only  necessary  to  make  the  indicated  minor  adjustments  in  reported  values. 
Therefore,  it  is  contended  that  the  main  findings,  as  presented  in  the  section 
on  results,  should  be  taken  at  face  value  so  far  as  differences  among  training, 
test,  and  retraining  performances  are  concerned,  with  minor  qualifications  added 
about  probabilities  to  be  expected  in  a  real  mission. 
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Accordingly*  the  considered  findings  arc  as  follows* 

1.  Original  Capability s 

a.  By  the  end  of  five  weeks  training  the  pilots  and  crews  achieved 
varying  levels  of  reliability  in  performing  the  nine  aiesicn  phases  according 
to  hypothetical  system  criteria. 

b.  The  first  two  crews  tented  (0-4  and  0-9)  wore  generally  such  less 
able  to  meet  the  criteria  than  the  second  two  orews.  Highly  reliable  (.999) 
performance  was  seldom  demonstrated  by  them  on  any  portion  of  the  mission  end 
even  the  best  performer  among  them  showed  a  likelihood  of  less  than  .850  of 
performing  within  the  criterion  on  any  parameter  (p).  The  likelihood  of  meeting 
all  criteria  (i.e. ,  £  or  mission  success)  was  never  more  than  .002.  Because  these 
crews  evidently  were  not  fully  trained,  their  data  are  discounted  in  the  present 
e*  idy  as  incapable  of  supporting  useful  generalisations  about  skill  retention  in 
space  missions. 

o.  The  second  two  crews  (0-8  and  0-13)  attained  quite  high  indicated 
reliability  levels.  Estimated  likelihoods  of  meeting  the  criterion  on  any  para- 
meter  ($)  ranged  from  .932  up  to  .990  and  of  meeting  criteria  in  all  parameters 
(p)  frem  .294  to  .817.  However,  the  lower  two  sets  of  values  (for  P-132  and  P-133) 
are  considered  underestimates  due  to  the  inclusion  of  unrepresentative  scores, 
whereas  all  other  values  are  considered  overestimates  of  true  capability  in  a 
real  mission  cf  this  nature.  The  extent  of  this  overestimation  is  Judged  to  b© 
in  the  order  of  .005  in  likelihood  of  meetir-  any  particular  criterion  and  of 
.10  in  likolihood  of  meeting  ail  criteria,  .ven  these  apparently  well-trained 
pilots  differed  somewhat  in  capability  in  the  several  measured  aspects  of  flight 
control,  the  different  phases,  and  the  overall  mission. 

2.  Test  Mission  Capability* 

a.  In  the  test  mission,  performed  approximately  8  to  13  weeks  after 
training,  the  well-trained  crews  (0-8  and  0-13)  showed  an  estimated  capability 
to  meet  the  criteria  somewhat  lower  than  their  original  capability  to  moot  them. 
The  first-tested  pilots  evidenced  considerable  loss,  but  some  of  those  tested 
second  and  third  showed  a  slight,  probably  spurious,  gain — the  orew  averages, 
consequently  indicating  mild  losses.  As  was  expected,  the  same  pilots  showed 
even  less  likelihood  of  reaching  former  highly  reliable  (p.95)  levels  of  perfornr- 
onoa  in  individual  parameters.  Evidently,  lack  cf  direct  practice  over  a  period 
of  8  weeks  or  more  resulted  in  a  definite  loos  in  skill  which  was  to  a  consider¬ 
able  extent  recovered  by  the  opportunity  to  observe  or  participate  once  or  twice 
in  another  pilot's  performance  of  tbe  mission.  However,  a  difference  in  retention 
over  8  and  13  weeks  was  not  demonstrated — perhaps  because  of  differences  in 
original  learning — which  indicates  that  such  variations  in  retention  intervals 
that  long  may  be  of  little  significance.  The  levels  of  reliability  derived  from 
tbe  first  test  performance  of  around  .92  of  meeting  the  oriterion  in  any  parameter 
and  of  around  .02  of  meeting  the  criteria  in  all  parameters  are  considered  to 
slightly  overestimate  real  mission  performances  under  like  circumstances  of 
previous  training,  intervening  activities,  etc. 

b.  The  incompletely  trained  crews  ( 0— 4  and  0-9),  tested  after  4  or  9 
weeks,  showed  generally  parallel  effects  of  the  lack  of  practice  on  capability  to 
meet  criteria— -but  to  a  lesser  extent  snd  somewhat  inconsistently.  This  differ- 


ones  is  attributed  directly  to  their  smaller  mat  of  original  skill.  0 ©waver, 
they  chowad  about  the  ease  loss  in  capability  to  perform  at  thsir  am  former 
levels  of  high  reliability  ©s  th®  well— trained  orswa.  This  suggests  the  interest¬ 
ing  possibility  that  losses  in  relative  reliability  (i.e.,  probability  of  rsalis- 
isg  forser  levels  of  highly  reliable  performance)  nay  bs  little  influenced  by 
original  skill  attainments, 

3.  Capability  with  Additional  Trainings 

a.  In  the  course  of  3  days  of  additional  training  the  trail-trained 
pilots  of  0»8  and  0-13  again  abased  3llghily  varying  oapabilitiea  in  the  several 
measured  aspects*  of  flight  control,  th®  different  phases,  and  the  overall  mission* 
They  tended  to  bs  less  able  to  mat  the  criteria  for  separation  end  doorbit,  brake 
and  hover,  and  earth  entry  phases  than  they  sere  ahl®  to  seat  the  criteria  for 
transposition,  lunar  orbit  and  docking.  Sven  in  the  vary  first  four  trials  of 
retraining  four  of  those  pilots  approximated  their  original  capabilities  for 
nesting  the  criteria  and  in  their  best  four-trial  blocks  in  the  various  phases 
all  six  exceeded  those  capabilities.  Thus,  original  capability  to  nset  system 
criteria  mo  quickly  regained.  That  the  original  capability  to  amt  criteria 

was  improved  upon  implies  (especially  because  retraining  values  axe  believed  to 
bo  underestimated)  that  these  pilots  had  not  originally  reached  tho  Halts  of 
their  potential.  Soaevor,  even  at  their  beet,  these  pilots  did  not  demonstrate 
in  retraining  their  full  original  capability  to  perform  all  the  aeasurod  aspects 
of  flight  control.  Their  efforts  to  msdnica  performance  -eith  respect  to 
criteria  with  accompanying  frequent  shifts  between  training  end  retraining  in 
relative  performance  on  aspects  within  phases,  along  with  th©  sensitivity  of 
this  analysis  to  such  shifts,  is  considered  the  probable  explanation.  Such 
changes  in  strategy,  sMch  are  costly  of  overall  reliability,  are  not  reflected 
in  tho  criterion-reference  analysis  because  they  are  generally  beyond  the 
arbitrary  sariEua  diBcriainabla  reliability  of  .999* 

b.  In  the  course  of  two  to  eight  additional  training  trials  on  a 
single  day,  the  less  sell— trained  pilots  of  0-4  sad  0-9  wore  able  to  demonstrate 
capabilities  for  Electing  the  systaa  criteria  comparable  to  or  exceeding  their 
original  capabilities  for  nee  ting  then.  However,  like  the  pilots  of  0-8  and 
0-13,  they  did  not  demonstrate  their  full  original  capability  to  perform  all  th® 
se&surad  aspects  ox  flight  control ,  although  they  nay  have  coae  somewhat  closer 
to  doing  eo.  They  continued  in  retraining  to  be  distinctively  less  able  to  soot 
the  criteria,  even  though  they  performed  at  least  as  well  relative  to  their  own 
former  levels.  Thus,  although  operating  at  lower  levels  of  capability  with  respect 
to  criteria,  the  pilots  of  0-4  and  0-9  responded  to  additional  training  in  about 
the  earn  way  as  did  the  pilots  of  0-8  and  0-13. 
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BTFLICmaSS 


The  detailed  out coses  of  the  study  just  givsa  are,  of  oouras,  valuable  in 
their  asm  right c  But,  of  even  greater  interest,  perhaps,  are  the  implications 
they  carry  specifically  for  the  design  of  space  systems,  and  sore  generally  for 
the  measurement  and  prediction  of  human  perforcencco  They  are  also  3u®jeative 
of  directions  in  traioh  additional  research  on  skill  retention  end  on  related 
measurement  methodology  should  go.  These  apparent  iaplioations  will  be  briefly 
presented,  in  turn,  in  this  concluding  section. 


gML.§2EM.Mte 

As  described  at  the  beginning  of  this 

report,  the  primary  intent  ©f  the  study  tsas  to  evaluate  the  degree  to  wfciok  skill 
in  critical  tasks  might  be  retained  in  space  missions  of  extended  duration.  For, 
in  designing  such  systems  one  would  certainly  like  to  know  whether  forgetting  on 
the  part  of  the  operators  would  be  enough  to  degrade  the  probability  of  mission 
success  below  acceptable  levels.  If  forgetting  in  that  amount  were  predicted 
then  an  obvious  doaign  strategy  would  be  to  seek  to  reduce  or  avoid  the  degrada¬ 
tion,  possibly  by  the  modification  of  the  mission  profile  or  the  mission  tasks  or 
by  the  use  of  special  training  procedures.  But,  it  is  first  necessary  to  know 
what  her  skill  retention  is  likely  to  be  a  problem. 

The  findings  of  the  present  study  indicate  that  a  carefully  qualified  or 
guarded  position  on  the  skill  retention  question  is  in  order.  If  the  mission 
test  performances  of  F-8l  and  F-131  (the  critical  test  personnel )  are  taken  at 
face  value  then  it  must  be  concluded  that  skill  retention  difficulties  oan 
indeed  occur.  P-81,  tested  after  8  weeks,  was  able  to  perform  most  of  the  tasks 
required  within  the  nominal  criteria,  but  he  did  not  meet  them  in  two  parameters. 
P-131,  tested  after  13  weeks,  also  was  able  to  perform  most  tasks  within  criteria, 
but  he  deviated  greatly  from  them  in  three  of  four  critical  parameters  in  the 
lunar  landing.  (On  a  real  mission  he  would  have  sheared  the  landing  gear,  at  the 
very  least.)  Because  of  these  difficulties  both  pilots*  estimated  reliabilities 
in  meeting  criteria  dropped  noticeably  from  their  former  levels  to  reliabilities 
which  would  be  unacceptable  for  a  real  mission.  (P-8l  dropped  6$  to  .931  in  p, 
14.^>  to  .827  in  i>  and  93-5$  to  .048  in  $  while  P-131  dropped  7.8$  to  ,899  in  P» 
9.3%  to  .858  in  p  and  100$  to  <.0C1  in  $.)  The  estimated  probabilities  of  mission 
success  (i.e. ,  of  meeting  all  criteria)  by  not  exceeding  five  obsnoes  in  a  thousand 
are  particularly  discouraging.  Quite  evidently  the  requirement  in  a  space  mission 
to  ramsmbor  a  critioal  flight  control  task  over  8  weeks  or  more  poses  a  problem 
worthy  of  special  design  attention. 

However,  in  addition  to  demonstrating  the  reality  of  tha  skill  retention 
problem,  the  present  data  provide  sues  insights  on  the  severity  of  the  problem 
and  indications  as  to  how  it  might  be  handled.  Of  particular  interest  in  this 
connection  are  the  teat  mission  performances  of  the  pilots  tested  second  and 
third  and  tha  retraining  performance  of  P-131.  (The  retraining  data  from  B-8l 
are  discounted  for  reasons  given  oar.lior.)  Although  pilots  82  ,  83,  132  and  133 
had  at  the  end  of  training  attained  approximately  the  ea bo  level  of  skill  in 
meeting  ths  criteria  os  P— 3l  and  P-131,  all  tut  P-82  showed  a  slight  gain  in 
test  over  training  and  P-82’8  loss  was  relatively  small.  Consequently,  there 
ssects  little  doubt  that  immediately  prior  observation  or  aiding  of  another  onoe 
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or  twice  in  performing  the  critical  tasks  can  bring  about  a  considerable  improve¬ 
ment  in  performance  after  8  or  more  weeks  without  direct  praotioe.  Furthermore, 
in  just  four  practice  trials  following  the  teat  mission-  P-131  equaled  or  exceeded 
his  own  former  capability  to  meet  the  nominal  criteria  (at  least  to  a  probability 
of  .999)  in  six  representative  phases  of  the  mission.  On  the  assumption  that  the 
chosen  criteria  are  reasonable  ones  for  such  systems ,  both  of  these  findings  imply 
that  the  skill  retention  problem  need  not  b©  severe.  They  suggest  that  the  deg¬ 
radation  in  mission  reliability  associated  with  recall  over  at  least  13  weeks  can 
largely  be  avoided  in  instances  where  intervening  activities  serve  to  maintain 
as  alertness  ;o  requirements  for  time-sharing  and  rapid  response  and  when  opportu¬ 
nities  exist  to  practice  or  even  rehearse  and  observe  the  critical  activities 
before  they  are  performed  in  the  mission.  Apparently,  the  preexposure  to  the 
tasks  need  not  be  extensive,  but  it  must  be  mors  than  merely  an  examination  of 
typical  handbooks  and  checklists  and  verbal  discussion. 

The  cements  and  discussion  of  the  debriefings,  conducted  informally  after 
the  test  mission,  lend  further  support  to  this  view.  In  essence,  the  pilots’ 
comments  then  (and  earlier)  emphasised  the  desirability  from  their  point  of  view 
of  an  opportunity  to  enter  the  crew  station  beforehand  and  refasdliariae  them¬ 
selves  more  directly  with  the  displays  and  oontrols.  They  felt  this  would  have 
alleviated  considerable  misgiving  on  their  part  about  performing  the  tasks 
adequately.  Beyond  this,  in  response  to  direct  questioning,  the  pilots  indicated 
a  belief  that  the  prior  opportunity  to  observe  or  rehearse  the  more  difficult 
phases  would  be  very  helpful  and  the  second-  and  third-tested  pilots  confirmed 
this  from  their  own  experience.  Host  agreed  that  the  casually-suggested  possibil¬ 
ity  of  a  preview  by  means  of  an  over-the-shoulder  filming  of  action  of  the  more 
complicated  phases  (perhaps  narrated  themselves  for  their  own  use)  might  be  quite 
helpful.  However,  P-*8l  took  exception  to  this  baoauBS  he  felt  that  only  the 
opportunity  to  actually  operate  the  controls  would  be  helpful  to  him. 

Of  particular  interest  in  the  debriefings  were  the  comments  of  P-131,  about 
whose  difficulty  discussion  naturally  centered.  He  stated  that  he  had  proceeded 
with  the  braking  mid  hover  sequence  very  cautiously  in  an  effort  to  do  well  and 
did  not  realize  until  too  late  to  fully  recover,  the  implications  of  his  exces¬ 
sively  alow  rate  of  descent.  He  described  this  as  a  problem  not  encountered  in 
training,  thus  seeming  to  imply  training  inadequacy,  and  said  that  now  being 
aware  of  such  a  possibility  he  would  not  likely  repeat  his  mistake.  However,  a 
somewhat  different,  but  not  contradictory,  interpretation  is  possible,  for,  it 
may  well  bo  that  P-131  did  not  encounter  the  problem  in  training  because  at.  that 
time  his  greater  awareness  of  the  timing  factors  involved  and  familiarity  with 
the  time-sharing  requirements  did  not  lead  him  to  it.  Perhaps  it  was  the  for¬ 
getting  of  just  these  task  features  which  occasioned  the  problem.  Certainly  that 
interpretation  is  in  keeping  with  some  of  the  other  pilots'  comments  on  the 
retention  test  as  well  as  vith  laboratory  studies  on  the  forgetting  of  oomplex 
motor  tasks  (Trumbo  et  al,  19^5)* 

Thus,  regarding  the  question  of  skill  retention  in  space  system  design,  it 
seems  fair  to  conclude  from  the  present  study  that  without  special  design  atten¬ 
tion  operator  reliabilities  in  critical  tasks  performed  after  8  weeks  or  more 
without  prior  direct  familiarization  are  likely  to  be  unacceptable.  This  does 
not  moan  failure  to  meet  criteria  necessarily  will  occur*— only  that  it  is  unaccept¬ 
ably  likely  to  ooour.  But,  if  judicious  use  is  made  of  training  techniques  prior 
to  the  more  difficult  critical  task  performances  which  afford  a  refamiliarization 
with  the  task  (particularly  the  time-sharing  and  timing  requirements)  much  of  the 
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expected  degradation  in  reliability  can  be  avoided.  However,  this  should  not 
be  taken  to  mean  that  the  full  original  capabilities  of  the  operators  will  be 
regained  by  such  expedients— in  fact,  the  present  data  suggest  that  muck  more 
would  b©  required  for  that.  Furthermore,  this  conclusion  rests  directly  on  the 
assumption  that  the  intervening  activities  required  do  not  give  rise  to  specific 
interference  effects  (negative  transfer  to  the  tasks  of  con carat).  Accordingly, 
the  possibility  of  adverse  transfer  from  intervening  activities  along  with  the 
choice  of  appropriate  refamiliarisation  techniques  must  bo  given  close  attention 
in  design  if  the  problem  of  skill  retention  in  long  missions  is  to  be  avoided. 

In  short,  skill  retention  will  not  be  a  problem  if  proper  attention  is  given  to 
skill  retention  requirements  in  design  and  operational  planning. 

Pilot  Reliability.  In  addition,  and  incidental  to  the  main  concern  for 
skill  retention,  the  study  also  provides  the  possibility  of  insights  into  pilot 
reliability  beyond  the  report  by  Grcdsky  et  al  (1966b)  of  the  original  HASA- 
Martin  study.  This  results  from  two  factors*  the  analytic  methodology  employed 
and  the  evident  gains  in  retraining  over  the  original  training  performance.  In 
the  retraining  portion  of  this  skill  retention  test  the  pilots  were  at  their 
vary  best  and  the  present  analysis  characterises  that  best  performance  in  a  way 
that  is  quite  different  from  the  method  used  by  Grodsky  et  al.  What,  than,  do 
the  results  obtained  indicate  about  pilot  reliability? 

If  it  is  assumed  once  again  that  the  hypothetical  oriteria  used  are  reason¬ 
able  for  space  systems — then  a  fair  conclusion  on  the  basis  of  the  retraining 
values  reported  for  0-8  and  0-13  in  table  XIII  should  be  possible.  Taken  at  face 
value  the  indicated  reliabilities  are  reassuringly  high,  for  among  the  sir  pilots 
the  minimum  values  obtained  are  ,991  in  p,  .964  in  i>,  and  .?89  in  p.  Obviously 
the  chances  of  performing  a  parameter  or  a  phase  successfully  (i.e. ,  without 
exceeding  criteria)  seem  quite  good  and  even  the  likelihood  of  performing  the 
whole  set  of  phases  successfully  seems  almost  aa  goo£  (the  second  lowest  f>  is 
.928).  The  average  probability  of  overall  success  (p)  for  the  six  is  .936. 
Furthermore,  the  evaluation  of  possible  biases  suggested  that  these  values  some¬ 
what  underestimate  reliability  in  a  real  situation. 

However,  there  is  another  factor  so  far  undent Done a  which  tends  to  inflate 
these  values,  particularly  for  p,  beyond  vhat  might  be  expected  in  a  full  mission. 
After  all,  these  probabilities  for  retraining  are  baaed  upon  only  six  of  the 
mission  phases  and  do  not  represent  a  oomplote  mission.  Inclusion  of  the  rest 
of  the  mission  phases  would  tend  to  lower  mission  reliabilities  even  though  para¬ 
meter,  and  possibly,  phase  values  could  be  increased  by  very  good  performance. 

It  xs  an  inherent  feature  of  joint  probabilities  that  when  other  factors  are 
added  the  product  can  only  equal  or  become  less  than  the  original  j  it  can  never 
exceed  the  original.  Thus,  while  difficult  to  estimate,  the  net  result  may  be 
that  the  indicated  overall  values  (p,  £>,  and  p)  are  very  close  to  what  might  be 
observed  in  a  full  mission.  Is,  then,  a  probability  of  around  .936  of  mission 
success  (f>)  high  enough  for  a  space  mission?  Some  may  doubt  that  it  is  high 
enough  and  appropriately  wonder  whether  pilot  reliabilities  in  lengthy  and 
involved  missions,  such  as  a  lunar  landing  mission,  are  likely  to  be  disappoint¬ 
ingly  low  and  to  pose,  when  taken  jointly  with  equipment  reliabilities,  a  worri¬ 
some  (or  even  grave)  risk  of  mission  failure.  In  view  of  present  planning  for 
space  systems,  to  have  to  answer,  Yes!  would  be  chagrining  indeed. 
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Fortunately,  a  negative  answer  seems  most  appropriate  for  two  rsaacms. 

First,  it  is  doubtful  if  the  pilots  of  0-8  and  0-13  at  their  boat  had  reached 
the  full  limits  of  their  potential.  The  still  considerable  variability  evident 
in  their  performance  as  well  a a  informal  teats  of  the  training  decision  rule 
(referred  to  earlier)  on  other  data  suggest  that  under  a  still  more  stringent 
training  regime  ( including  revised  task  goals  and  feedback  on  performance)  all 
of  these  pilots  could  reach  ntrc  highs  in  performance  excellence.  Operational 
training  may  be  or,  at  least,  can  be  superior  to  that  given  the  test  personnel 
in  this  study. 

Second,  the  so  far  unquestioned  assumption  as  to  the  reasonableness  of  t'm 
criteria  used  say  still  be  doubted  with,  perhaps,  good  reason.  It  is  not  ao 
much  that  a  question  can  be  raised  about  one  or  more  of  the  actual  values  used* 
they  can  be  easily  remedied  V  the  justification  and  choice  of  new  values,  should 
anyone  o&re  to  do  so.  Esther,  the  possible  problem  with  the  criteria  earns  to 
come  directly  from  the  very  process  of  using  then  at  all.  ¥ith  respect  to  many 
aspects  of  flight  control  setting  a  criterion,  other  than  an  extremely  broad  one, 
is  an  arbitrary  process.  The  characteristics  of  the  machine  with  which  the  opera¬ 
tor  interacts  most  comcsnly  ore  such  that  the  effects  of  Ida  error  vary  contin¬ 
uously  in  magnitude  over  quit©  a  range  before  catastrophic  degradation  in  effective¬ 
ness  occurs.  The  overotresB  point  is  often  wall  beyond  any  human  performance 
criterion  of  interest  and  so  frequently  the  criteria  used  can  ha  exceeded  without 
undue  sacrifice  in  mission  accomplishment  and  an  appropriate  recovery  can  be  made. 

In  the  present  instance  it  seems  evident  that  within  sene  limits  most  of  the 
typical  criteria  used  could  bs  exceeded  without  mission  failure  ocourring-~«xo0pt 
in  the  narrow  sense  of  not  meeting  all  criteria.  For  example,  a  AV  error  of 
11  fps  is  not  appreciably  sore  significant  than  a  AV  error  of  9  fps,  mid  so 
forth.  Consequently,  reliabilities  with  respect  to  such  arbitrary  criteria  must 
be  taken  with  due  regard  to  their  actual  implications  for  ths  mission  as  well  as 
their  face  value.  Probabilities  of  sissies  success  rather  lower  than  ordinarily 
expected  may  not  b©  worthy  of  serious  concern  after  all. 5 

Thus,  in  keeping  with  ths  ambiguities  involved  in  specifying  reliability 
by  means  of  arbitrary  criteria  and  the  belief  that  the  pilots  of  0-8  and  0-13 
could  have  achieved  still  greater  reliability,  tha  present  data  should  not  occas¬ 
ion  doubts  about  pilot  capability  to  satisfactorily  perform  the  type  of  mission 
used  in  this  study*  ¥ith  carefully  arranged  additional  training  ths  performance 
reliabilities  attained  would  likely  exceed  those  calculated  from  retraining 
performance  in  this  study  and  on  close  scrutiny  they  would  bo  found  to  represent 
quite  acceptable  performance.  Howavor,  along  with  careful  attention  to  the  train¬ 
ing,  periodic  assesenent  of  performance  reliabilities  would  be  necessary  to  assure 
this  happy  result — either  in  a  simulated  or  in  a  real  mission.  The  attenuation 
of  mission  effectiveness  as  missions  become  more  lengthy  a^d  involved  necessitates 
more  than  a  oasual,  offhand  approach  to  training  and  a  quantitative  validation 
rather  than  an  informal  judgment  of  performance  capability. 


This  argument  is  also  applicable  to  the  eonoom  for  skill  retention  and  so 
reduces  the  severity  of  the  problem.  But  it  does  not  eliminate  the  problem 
completely  or  the  need  for  special  attention  to  it  in  system  design. 
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In  addition  to  thee©  specific  implications  for  space  system  design,  this 
study  (score  particularly  the  analysis  presented)  also  illustrates  boss  novel 
methods  for  handling  human  performance  data.  Bo  cause  this  is  the  first  formal 
application  of  these  methods  a  few  general  ©assents  about  their  utility— both 
for  the  measurement  and/or  prediction  of  misaa  reliability  in  system  development 
and  operational  planning  and  for  the  quantification  of  behavior  gansraliy—seem 
in  order. 

f±ntf  the  out¬ 
come  of  the  analysis  has  a  familiar  ring.  Not  only  are  the  reliabilities  obtained 
surprisingly  compatible  with  general  ifspressions  of  pilot  capability  but  that 
expression  of  capability  is  familiar  to  systems  engineers.  For,  in  anticipating 
the  effectiveness  of  systems  under  design  and  development,  reliability  studies 
estimating  mean  time  between  failure  and,  in  turn,  likelihood  of  successful  opera¬ 
tion  in  the  single  case  are  required.  Only  when  such  information  is  available 
can  appropriate  cost-effectiveness  trade-offs  be  made  and  maxitaus  functional 
value  be  obtained  from  the  system.  Furthermore,  subsequent  operational  planning 
must  or  should  be  prefaced  upon  detailed  ■'nforasation  on  specif io  capabilities. 

However,  even  though  this  approach  to  system  development  has  bean  widely 
discussed  and  is  now  well-accepted  th©  full  implementation  of  it  with  respect  to 
all  critical  performances  within  the  system  has  not  and  is  not  now  being  accom¬ 
plished.  is  has  long  bean  recognized  by  human  factors  specialists  and  others, 
ths  capabilities  of  the  people  Involved  (either  as  operators  or  servicers)  have 
a  considerable  influence  on  effectiveness  (see,  for  example,  Shapsro  et  al,  i960 
and  Thomas,  1962).  Yet  studies  of  system  effectiveness  have  continued  to  dis¬ 
count  the  element  of  human  performance  in  favor  of  preoccupation  with  the  more 
familiar  and  understandable  machine  elements.  Machine  reliability  is  required 

for  system  effectiveness — but  so  is  human  reliability  in  most  systems  of  Interest. 

*> 

jSaong  the  several  reasons  for  this  persistent  disregard  of  human  perform¬ 
ance  in  system  design  the  reason  of  particular  relevance  here  (and  perhaps  most 
important  of  oil)  is  simply  that  human  performance  data,  when  available,  are 
usually  not  in  a  form  amenable  to  direct  use  in  effectiveness  studies.  In  pre¬ 
dicting  on  a  particular  system  what  one  needs  to  know  is  what  are  the  probable 
limits  of  this  individual  *  e  (or  this  select  group  of  individuals*)  performance 
of  these  particular  activities.  The  average  or  most  typical  performance  of  a 
general  sampling  of  individuals  on  a  collection  of  traditional  laboratory  tasks 
is  not  of  immediate  use  and  seemingly  foreign  to  the  design  problem.  Further¬ 
more,  the  necessary  interpreter  of  such  data  is  often  not  available.  In  contrast, 
the  analytic  methods  of  this  study  offer  the  special  merit  of  placing  these  human 
performance  data  in  terras  of  immediate  utility  and  that  is  why  the  results  obtained 
seem  familiar  (to  the  systems  engineer! ). 

This  gratifying  re  stilt  is  primarily  the  consequence  of  the  fundamental! 
innovation  of  interpreting  observed  performance  probabilistically,  liter  all, 
human  performance  is  variable  (to  the  great  dismay  of  psychologists  who  have 
sought  to  express  invariant  relations  involving  it)  and,  therefore,  stateawtats 
about  it  are  probabilistic,  whether  the  fact  is  recognised  or  not.  To  express 
performance,  then,  directly  as  «.  probability  is  merely  to  explicitly  reocanise 
the  variability  which  does  exist.  To  do  so  is  to  stay  desirably  close  to  observa¬ 
tional  realities  and  avoid  preoccupation  with  finding  supposed  ''true"  values,  and 
so  forth. 
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How  the  expression  of  perforsanca  aa  a  probability  raquiree  that  some 
reference  b©  chosen— i.®. ,  that  ths  probability  b®  with  respect  to  something. 

Two  sain  alternatives  are  possible  and  they  are  both  exemplified  in  the  analysis 
of  this  study  data*  One  nay  either  refer  to  no m  absolute  criterion  dictated  by 
outside  considerations  (system  requirements,  etc.)  or  one  say’  refer  to  some  rel¬ 
ative  criterion  established  by  the  individual’s  own  or  acme  other  individual’s 
performance.  The  first  option  is  straightforwardly  an  application  of  the  proV 
ability  concept  which  relates  human  performance  to  ths  requirements  for  it  in 
such  a  manner  as  to  directly  express  capability  of  meeting  those  requirements. 
Consequently,  it  is  extremely  useful  in  making  design  choices  as  wall  as  assess¬ 
ments  later  of  operational  capability. 

On  the  other  hand,  the  second  option  provides  an  extremely  sensitive  method 
of  indicating  change  or  difference  consequent  upon  a  change  in  conditions  or 
personnel.  One  simply  chooses  an  appropriate  probability  and  establishes  that 
level  or  standard  of  performance  in  the  units  of  measure  for  the  reference.  Ths 
probability  estimated  from  any  other  performance  of  achieving  that  standard  level 
may  then  bo  directly  compared  with  the  reference  probability  and  the  amount 
(absolute  or  relative)  of  shift  in  probability  taken  to  indicate  ths  magnitude 
of  the  effects.  In  this  way  performance  change  is  described  probabilistically 
rather  than  in  terms  of  the  arbitrary  units  of  measure  (suoh  as  feet  per  tec. , 
degrees,  etc.).  This  option  has  special  value  for  empirical  determination  of 
the  relative  merits  of  two  or  more  design  arrangements  when  outside  criteria 
either  do  not  exist  or  do  not  readily  discriminate  among  ths  arrargements.  But 
in  many  cases  the  information  parovid/ad  by  reference  to  both  absolute  and  relative 
criteria  is  needed — as  was  true  in  the  present  study. 

In  expressing  relative  reliability  it  is  neoessaxy  to  choose  a  referenoe 
probability  and  so  it  is  naturally  of  concern  what  choioe  is  best  or  bow  the 
choice  should  be  made.  In  this  instance  p  a  ,950  was  chosen,  but  why  was  this 
value  rather  than.  say.  p  «  ,500  (the  mean  of  a  normal  distribution)  selected? 

The  most  reasonable  answer  is  that  there  is  no  one  best  value  for  all  circum¬ 
stances  but  rather  a  need  to  choose  ths  value  on  the  basis  of  the  immediate 
purpose.  In  the  present  case,  the  interest  was  in  performance  levels  attainable 
at  a  high  level  of  consistency  end  so  the  *550  was  selected*  Ihixthormore,  it 
is  usually  true  that  what  one  is  interested  in  is  the  performance  achievable 
with  near  certainty  (or  else  that  islaoat  certainly  not  achievable)  rather  than 
merely  the  most  typical  performance.  This  suggests  that  a  reference  in  the  order 
of  p  «  .950  would  not  generally  be  amiss.  But,  circumstances  dictating  an 
interest  in  the  most  typical  (p  ■  .500)  or  in  some  intermediate  value  (suoh  as 
P  -  .750)  can  readily  be  imagined.  Whatever  the  choioe,  the  essential  point  to 
be  emphasized  here  is  tkas  in  describing  performance  probabilistically  this 
significant  choice  for  any  kind  cf  statistical  analysis  is  made  explicit.  In 
this  there  is  great  merit  because  the  traditional  practice  of  describing  perform¬ 
ance  in  terms  of  central  tendency-discounting  variability  as  it  does — deserves 
justification  each  time  it  is  used. 

Finally,  in  depicting  perfor, nance  as  a  probability  a  very  considerable 
advantage  is  realized  in  the  handling  of  realistic  behavioral  complexities  such 
as  typically  are  found  in  operational  systems.  One  of  the  great  faults  of  con¬ 
ventional  psychological  methods  is  they  do  not  provide  a  satisfactory  means  of 
describing  complex  performances  vh.Loh  are  characterised  by  two  or  more  measures 
(muoh  less  22,  as  in  this  study).  Yet,  in  reality,  it  is  an  inclusive  expression 
of  the  totality  of  a  performance  which  is  most  often  desired.  Sven  the  normalizs- 
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tier.  and  subsequent  combination  (weighted  or  not)  of  the  several  data  series 
does  not  rsaody  this  difficulty,  for  what  results  is  merely  an  index  relative  to 
the  particular  data  collection  circumstances  and  having  no  independent  moaning. 

In  contrast,  the  expression  of  performance  as  a  probability  not  only  similarly 
permits  a  combination  of  the  several  measured  aspects  to  arrive  at  a  joint  indi¬ 
cant  but  provides  statements  of  probability  to  trhioh  other  events  say  be  related 
(as  within  a  system).  The  result  is  that  analysis  in  terms  of  probabilities 
allows,  whert,  conventional  methods  do  not,  the  modeling  of  a  complex  series  of 
aanr-m&ehino  functions  necessary  to  effectiveness  studies. 

Thus,  it  is  concluded  that  the  analytic  methods  employed  in  this  study  also 
constitute  valuable  techniques  for  introducing  human  performance  considerations 
into  the  design  of  systems.  By  describing  performance  probabilistically  the 
advantages  of  staying  close  to  the  realities  of  the  data  and  of  using  terms 
familiar  to  many  design  personnel  are  realized.  Concern  for  human  performance 
is  encouraged,  thereby,  as  it  should  be.  By  interpreting  performances  as  the 
likelihood  of  meeting  absolute  criteria,  derived  from  system  analysis,  the 
opportunity  to  influence  specific  design  choices  as  well  as  to  assess  operational 
capability  with  respect  to  human  performance  is  realized.  For  example,  simula¬ 
tion  tests  of  actual  skill  retention  requirements  can  be  properly  evaluated  and 
the  effectiveness  of  proposed  remedial  techniques,  when  required,  can  be  checked. 
By  interpreting  performance  as  the  likelihood  of  meeting  relative  criteria, 
derived  from  the  probabilistic  description  of  another  performance,  an  extremely 
sensitive  indication  of  differences  or  charges  in  perfc.manoe  is  obtained.  This 
is  a  powerful  technique  for  exploring  particular  design  issues  and  guiding  opera¬ 
tional  plans  concerning  training  and  other  details.  For  example,  a  simple  exten¬ 
sion,  such  as  the  training  decision  rule  alluded  to  earlier,  oan  be  used  as  the 
basis  for  realistic  quantitative  evaluation  of  achievement  in  training;,  thus 
eliminating  personal  judgments  as  the  primary  basis  for  training  decisions.  This 
would  assure  the  desired  human  reliability  in  complex  and  lengthy  missions  by 
more  precise  determination  of  day  to  day  training  needs  and,  in  turn,,  more  effec¬ 
tive  training.  Finally,  by  expressing  human  performance  in  various  aspects  of 
the  job  probabilistically  the  desirable  option  of  formulating  statements  of  over¬ 
all  probability  and  of  entering  human  performance  in  system  effectiveness  models 
is  gained.  These  are  some  of  the  more  significant  and  obvious  merits  of  the 
analytic  techniques  used — but  not  necessarily  all  their  merits. 

But  what  about  the  other  side  of  the  ledger,  one  may  ask.  What  are  the 
drawbacks  and  disadvantages  of  these  methods?  From  a  practical  point  of  view 
it  should  be  noted  that  a  great  number  of  calculations  are  required— *a  tedious 
process  if  accomplished  by  hand  (as  for  this  study).  But,  in  any  settled 
application — aay  in  conjunction  with  mission  simulation — computerized  data 
reduction  would  be  simple  to  arrange  and  of  trivial  cost.  The  calculations 
required  are  Bimple  ones  and  in  themselves  rather  familiar,  at  least  when  the 
normal  probability  model  is  adopted.  The  novelty  involved  is  in  the  adaptation 
and  combination  of  these  basic  and  familiar  techniques.  On-line  calculation, 
as  might  ba  of  real  advantage  in  a  training  situation,  is  imminently  practicable. 
Furthermore,  although  the  full-blown  methods  and  associated  jargon  initially 
would  seem  strange  to  human  factors  specialists  trained  in  traditional  methods, 
their  natural  desire  to  see  human  performance  more  fully  considered  and  the 
familiarity  of  the  basic  concepts  should  dispose  the*,  to  quickly  grasp  the 
methods  and  apply  them,  as  required.  In  fact,  judging  from  present  trends,  the 
only  system  design  personnel  who  probably  would  not  readily  acoept  these  methods 
are  the  operational  personnel  (i.e. ,  the  research  pilots  involved)  who  are 
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generally  disinclined  to  have  numbers  placed  upon  their  performance*  Sven  this 
may  not  be  too  serious  a  drawback;,  for  if  a  suitable  trial  usage  is  arranged  and 
due  consideration  is  given  to  pilot  suggestions  aa  to  details  (measured  parameters, 
models,  etc#)  it  is  possible  that  many  such  individuals  would  bsooss  more  accept¬ 
ing#  For  when  used  in  the  proper  way  these  methods  avoid,  at  least,  some  of  the 
deficiencies  in  conventional  measurement  methods  which  have  prompted,  .justifiably, 
some  of  the  misgivings  of  operational  personnel  about  performance  measurement. 

But,  convinced  or  not,  the  quarrel  of  operational  personnel  is  likely  to  be  with 
measurement  per  se  rather  than  with  these  methods  and  that,  of  course,  is  more 
of  a  political  than  a  technical  issue# 

In  short,  there  sees  to  be  a  variety  of  advantages  and  virtually  ao  dis¬ 
advantages  attendant  upon  applying  the  analytic  methods  used  in  this  study  in 
system  davelopraent  and  operational  planning. 

Quant if ioat ion  of  Behavior.  But  what  about  the  still  broader  implications 
of  the  analysis  for  the  quantification  of  behavior,  apart  from  the  concern  for 
particular  systems?  If  the  methods  sire  useful  for  the  quantification/ prediction 
of  behavior  in  man-machine  ey -items,  then  perhaps  they  also  ought  to  bn  useful  in 
quantifying  behavior  in  other  circumstances,  as  in  laboratory  studies  having  no 
intended  specific  application.  And,  indeed,  it  i3  believed  that  this  expectation 
is  entirely  warranted#  Although  the  arguments  are  not  so  evident  why  psychological 
research  would  lac  benefit  greatly  from  considered  usage  of  these  methods  (and 
this  is  not  the  appropriate  place  to  detail  them)  they  parallel  to  a  great  extent 
with  elaborations  and  ramifications  those  already  given  concerning  systems  applica¬ 
tions.  Certainly  it  would  be  beneficial  and  desirable  to  place  research  data  in 
a  form  amenable  to  direct  generalization  (if  permissible  otherwise)  to  specific 
oases  if  this  can  be  accomplished  without  sacrifice  in  other  research  goals.  With 
these  methods  such  is  possible  and  in  this  way  the  bridge  between  the  laboratory 
and  the  world  of  applications  can  be  strengthened.  But,  apart  from  this  very- 
real  advantage  in  the  application  of  behavioral  research  there  are  also  advantages 
to  be  gained  in  the  way  of  improved,  more  effective  research.  A  more  complete 
statement  of  this  view  on  analytic  methodology  and  the  justification  for  it  may 
be  found  in  another  report  (Cotterman,  1967). 


Further  Reaearoh 

Finally,  in  addition  to  the  implications  for  apace  system  design  and  the 
measurement/ predict ion  of  human  performance  the  results  obtained  suggest  direc¬ 
tions  which  the  course  of  further  research  should  take.  There  are  many  research 
possibilities  which  could  be  mentioned,  but  only  the  seemingly  more  urgent  will 
be  considered  in  the  following. 

Skill  retention.  With  reapeot  to  skill  retention  oomplax  and  lengthy 
space  missions  it  certainly  cannot  be  asserted  that  this  study  provides  all  the 
needed  answers.  The  significance  of  the  concern  for  skill  retention  is  believed 
established  by  this  study  but  this  is  not  enough.  What  is  needed,  in  addition, 
is  refined  information  on  exactly  how  much  loss  is  to  be  expected  as  a  function 
of  time  periods  ranging  from  a  week  or  two  to  the  limit  of  the  longest  mission 
contemplated  in  the  foreseeable  future.  Furthermore,  this  relationship  should 
be  established  for  a  sufficient  number  of  diacriminably  different  kinds  of  tasks 
as  to  be  representative  of  the  anticipated  behavioral  requirements,  for  skill 
retention  is  known  to  depend  upon  the  type  of  task  involved.  To  accomplish  this 
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Kill  require  utmost  ear©  in  research  to  establish,  ssong  other  things,  that 
prior  to  teat  tha  psrsoansl  involved  have  achievad  known  and  equal  levels  of 
aohicrvQcsnt.  Tha  training  decision  rule  referred  to,  eaio&a  this  possible,  al¬ 
though  sees  further  development  of  it  before  application  is  desirable,,  Such, 
inforsatiern  mist  b®  available  if  accurate  prediction  on  the  magnitude  of  tbs 
skill  ret  ant  ion  problem  ever  is  to  bo  made  during  ths  course  of  design.  Without 
it  tha  only  recourse  is  to  an  empirical  determination  vie  simulation  in  oaoh  and 
every  instance. 

In  addition,  prediction  of  the  cagnituda  of  the  skill  retention  problem  is 
still  not  enough,  Again,  unless  a  quick  fix  in  each  instance  is  an  acceptable 
strategy,  tsaya  of  aspeliorating  tha  problem  should  also  ha  evaluated,  developed , 
and  refined  relative  to  the  range  of  behavioral  requirements  of  interest,  Beeearch 
having  these  goals  sight  range  all  the  way  from  specific  changes  in  task  require¬ 
ments,  through  various  training  manipulations  either  originally  or  during  the 
intervening  period,  to  consideration,  of  relevant  selection  factors.  Even  if  it 
mre  limited  to  training  remedies  a  programmatic  effort  likely  to  sake  serious 
headway  with  this  problem  would  have  to  be  extensive.  Obviously,  iaich  more 
research  on  the  skill  retention  problem  is  needed, 

HSigBBBBggl Jg&9&lSgfr  Similarly,  although  the  analytic  techniques 
employed  in  this  study  are  evidently  considered  to  represent  a  Bi scabies  step 
forward  in  capability  to  introduce  human  performance  measurement  in  system  design, 
development ,  and  analysis  and  a  contribution  to  behavior  quantification  generally, 
they  also  arc  not  complete.  Among  the  more  significant  needs  in  this  regard,  are 
the  followings  (l)  development  of  applications  with  the  use  of  other  than  the 
normal  probability  (or  tha  h)  model,  (2)  adaptations  and  oonf inaction  of  tests 
of  significance  for  application  to  probabilities  in  various  circumstances,  (3) 
more  intensive  analysis  and.  development  of  the  fundamental  implications  and 
rules  for  application,  and  (4)  development  of  more  refined  methods  of  modeling 
complex  events  (i.e. ,  events  haring  multiple  measured  aspects  which  say  be  at 
least  partially  interdependent).  As  further  work  of  this  nature  is  accomplished 
and  made  available,  the  scope  and  utility  of  this  novel  measurement  methodology 
will  be  greatly  enhanced,  with  bone fits  both  to  psychological  applications  and 
to  behavioral  research. 


SUMMARY 


Th®  primary  intent  of  this  study  was  to  obtain  a  nor©  valid  estimate  than 
eas  otherwise  available  of  the  degree  of  skill  loss  whioh  say  bo  expected  over 
tins  intervals  up  to  three  months  duration  in  tasks  typical  of  space  vehicle 
operation.  To  accomplish  this,  fcsir  crews  of  thro®  each  aerospace  research  pilots 
who  had  trained  for  a  period  of  six  weeks  in  the  performance  of  a  simulated  7~d ay 
lunar  landing  mission  were  tested  various  periods  of  time  after  the  conclusion  of 
training.  Each  oiow  was  tested  after  a  different  interval — the  intervals  being 
approximately  4»  8,  9,  and  13  weeks.  The  training  had  oulminsted  in  the  real -'time 
performance  of  the  7*~dsy  missions  however,  for  test  purposes  this  mission  was 
compressed  into  a  single  13-hour  workday  by  the  omission  of  long  ooast  phases,, 
navigational  tasks,  certain  secondary  activities,  ant  (for  two  orawa)  the  duplica¬ 
tive  transearth  insertion  phase.  By  this  economization  of  time  and  costs,  on® 
day  of  additional  training  for  the  4”  and  9~we®k.  crews  and  three  days  of  addi¬ 
tional  training  for  the  8-  and  13-week  orews  was  provided  as  a  means  of  checking 
u. on  how  quickly  former  skill  in  selected  phases  might  be  reacquired,  or  possibly 
surpassed.  In  the  test  mission  and  in  the  retraining  all  the  pilots  in  each  orew 
perforated  in  all  positions  (pilot,  navigator,  and  engineer)  so  that  data  wore 
obtained  from  each  of  the  twelve.  But,  with  the  exception  of  transposition  by 
4-  and  9-w'esk  orews,  the  commander  was  always  tested  in  the  pilot  position  first 
so  that  one  record  of  critical  tost  mission  performance  from  each  crow  would  be 
unoontaainated  by  prior  partioipatiou.  Srtenai  ,<*  porforssno©  record**  of  flight 
control  parameters  and  switching  activities  war®  obtained. 

The  analysis  of  the  performance  records  focused  atte.  .ion  upon  22  flight 
control  parameters  considered  critical  in  the  performance  of  the  nine  main  phases 
of  the  test  mission.  The  phases  reflected  by  these  are,  successively,  trana- 
lunar  insertion,  transposition  (of  modules),  lunar  orbit  insertion,  separation 
and  deorbit,  braking  and  hover  (lunar  landing),  lunar  powered  ascent,  nradosvous, 
docking,  and  earth  entry.  The  number  of  parameters  considered  descriptive  of 
any  given  phase  ranged  from  one  (for  two  phases)  to  four  (for  on®  phase),  The 
switching  data  axe  summarised  for  reference  but  because  of  their  nature  they  are 
discounted  as  contributing  little  in  this  study  to  the  understanding  of  skill 
retention. 

By  the  use  of  novel  analytic  methods,  extensive  analyses  of  the  flight 
control  performance  at  the  end  of  training,  in  the  test  mission,  and  in  i drain¬ 
ing  wore  accomplished  in  auoh  a  way  that  the  level  of  performance  observed  is 
given  as  a  probability  (or  reliability  value)  rather  than  in  the  units  of  measure. 
In  the  main  analysis  the  probabilities  make  reference  to  hypothetical  system 
criteria  for  the  22  parameters  and,  thus,  indicate  the  likelihoods  of  meeting 
those  criteria.  In  a  secondary,  parallel  analysis  of  test  and  retraining  perform¬ 
ance  the  probabilities  make  reference  to  the  level  of  performance  estimated  to 
be  achievable  by  eacl  individual  in  95$  of  his  performances  under  like  ciroua- 
stanoee.  thus,  indicating  the  likelihoods  of  meeting  those  former  levels  of  highly 
reliable  performance.  In  both  analyses  the  probabilities  of  meeting  individual 
parameters  within  given  phases  are  taken  cumulat ivoly  ^p)  as  an  estimate  of 
likelihood  of  success  in  any  parameter  and  jointly  (p)  as  an  estimate  of  likeli¬ 
hood  of  success  in  all  pareaetera  of  the  phase.  Similarly,  these  phase  prob¬ 
abilities  are  taken  cumulatively  as  estimates  of  success  in  any  parametor  (p)  or 
as  estimate?  of  smoco&s  in  any  phase  (p)  in  the  whole  mission  (or  a  collection 
of  poises  &  rn  retraining) ,  The  probabilities  of  success  in  all  parameters  of 
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r  phase  also  are  taken  joint,  iy  (p)  as  estimates  of  success  in  all  parameter a 
and  phases  of  the  whole  mission  (or  a  collection  of  phases)— l.e. ,  the  prob¬ 
ability  of  mission  success.  In  this  say  overall  performance  indicant e  for  each 
pilot  were  obtained. 

On  the  basis  of  ca.  ,ful  study  and  croee-oossparison  of  the  probabilities 
for  individuals  and  (taken  cumulatively)  for  crew  perforrnanoo  ae  well  as  the 
nature  mid  extant  of  possible  biases  involved,  a  number  of  specific  findings 
were  expressed.  With  regard  to  original  capability  at  the  end  of  training  the 
crews  tested  after  8  and  13  weeks  wore  found  to  have  achieved  quite  high  reli¬ 
abilities  ranging  for  individuals  from  ,932  to  ,990  in  meeting  any  criterion 
(p)  and  from  ,294  to  ,817  in  masting  all  criteria  (p).  However,  the  lower  two 
sets  of  values  are  considered  underestimates  due  to  *no  inclusion  of  unrepresenta¬ 
tive  scores,  whereas  all  others  are  considered  overestimates  of  true  capability 
in  a  real  mission  of  this  nature  (to  the  extent  of  about  ,005  in  p  and  ,10  in  p). 
Hence,  their  typical  likelihood  at  that  time  of  meeting  any  criterion  in  a  real 
mission  is  judged  to  have  been  around  .975  while  their  likelihood  of  meeting  all 
criteria  is  judged  to  have  been  around  .615*  In  contrast,  the  crews  tested  after 
4  and  9  weeks  were  found  to  have  had  much  less  capability  at  the  end  of  training 
(p  and  p  never  greater  than  .850  and  .002,  respectively).  Because  these  crews 
obviously  were  not  fully  trained  their  data  were  discounted  as  incapable  of 
supporting  useful  generalizations  about  skill  retention  in  space  missions  (al¬ 
though  analyses  of  them  are  included). 

In  the  test  mission,  performed  approximately  8  or  13  weeks  after  training, 
the  well-trained  crews  showed  an  estimated  capability  to  meet  the  criteria  some¬ 
what  lower  than  their  original  capability  to  meet  them.  The  first-tasted  pilots 
evidenced  considerable  loss  with  estimated  p  values  dropping  to  ,931  and  ,899  and. 
b  values  dropping  to  .048  and  <,001,  for  the  8-  and  13-week  pilot,  respectively, 
but,  some  of  those  tested  second  and  third  shoved  a  slight  (in  part,  spurious) 
gain — the  crew  averages,  consequently  indicating  mild  losses.  As  was  expected, 
the  same  pilots  showed  even  less  likelihood  of  reaching  former  highly  reliable 
(p.95)  levels  of  performance  in  individual  parameters.  Evidently,  laok  of  direct 
practice  over  a  period  of  8  weeks  or  more  resulted  in  a  definite  loss  in  skill 
which  was  to  a  considerable  extant  recovered  by  the  opportunity  to  observe  or 
participate  once  or  twice  in  another  pilot’s  performance  0?  the  mission.  However, 
a  difference  in  retention  over  8  and  13  weeks  was  not  demonstrated — perhaps  be- 
oause  of  differences  in  original  learning— -which  indicates  that  such  variations 
in  retention  interval  may  bo  of  little  significance  when  the  intervals  are  that 
long.  The  levels  of  reliability  derived  from  the  first  test  performance  of 
around  .92  of  mooting  the  criterion  in  any  parameter  and  of  around  .02  of  meet¬ 
ing  the  criteria  in  all  parameters  are  considered  to  somewhat  overestimate  real 
mission  performances  under  like  circumstances  of  previous  training,  intervening 
activities,  etc. 

Ir.  the  course  cf  three  days  of  additional  training  the  well-trained  pilots 
(those  tested  after  8  and  13  weeks)  quickly  regained  and  eventually  surpassed 
their  original  capability  to  meet  tt  Ttea  criteria.  Their  typical  probabilities 
of  successful  performance  in  a  r©a]  .on  at  this  time  ara  estimated  to  have 
been  about  .996  for  p  and  .936  for  t  rhis  implies  that  they  had  not  quite 

reached  the  limits  of  their  potential  in  original  training.  But,  even  at  their 
best,  these  pilots  did  not  demonstrate  in  retraining  their  full  original  capabil¬ 
ity  (relative  to  former  p  qc)  levels)  to  perform  all  the  measured  aspects  of 
flight  control.  Their  efforts  to  maximize  performance  with  respect  to  criteria 
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with  aeoospsayiag  froquani  shifts  batmen  training  and  retraining  in  relative 
psrfors&’-esa  on  aspects  within  phases,  along  with  the  sensitivity  of  this  anal¬ 
ysis  to  such  shifts,  is  considered  the  proha  bio  explanation.  Sash  changes  in 
strategy,  which  are  costly  of  overall  reliability,  ar®  not  reflected  la  the 
criterion-reference  analysis  because  they  arc  ganar&Xly  beyond  iha  arbitrary 
aariaaca  diacrlMnabla  reliability  of  ,$99« 

These  specific  findings  arc  considered  to  have  a  jatsbwr  of  valuable  implica¬ 
tions  for  space  system  design  and  for  the  asasu rsasnt  arid  prediction  of  tesan 
performance*  Thus,  it  is  concluded  with  respect  to  shill  retention  in  long 
spaos  missions  that  a  requirosent  to  ressabax-  a  critical  flight  control  task  8 
wssks  or  more  does  pom  a  probles  worthy  of  special  design  attention.  Be mover, 
assuming  the  intervening  activities  maintain  alertness  to  timesharing  &-id  rapid 
response  and  introduce  no  interference,  up  to  at  least  13  weeks  the  problem  is 
not  likely  to  be  so  severs  that  it  cannot  be  rose  died  by  judicious  use  of  train¬ 
ing  techniques  prior  to  critical  task  performances  for  ^familiarisation — 
particularly  with  lining  and  time- sharing  aspects.  More  refined  information 
than  this  study  provides  is  needed  to  allow  precise  prediction  about  skill 
retention  problems  and  planned  iessadiation  in  design.  With  respect  to  pilot 
reliability  in  space  missions  it  is  concluded  that  on  the  basis  of  these  data 
(maximl  capability  in  retraining)  no  doubts  about  pilot  capability  to  perform 
the  type  of  mission  used  in  this  study  are  justified  providing  art  res®  care  is 
given  to  training  and  to  obtaining  demonstrated  performance  reliability  in  the 
o curse  of  it. 

With  regard  to  the  measurement  and  prediction  of  human  performance  it  is 
contended  that  the  novel  analytic  methods  used  constitute  valuable  (and  needed) 
techniques  for  introducing  huaan  performance  considerations  in  the  design  of 
systems.  They  appear  to  offer  many  advantages  and  almost  no  disadvantages  in 
this  application.  Furthermore,  they  are  also  viewed  as  offering  a  large  poten¬ 
tial  use  in  the  quantification/  predict  ion  of  behavior  generally.  Accordingly; 
as  continuing  efforts  to  enhance  the  scope  and  utility  of  thia  novel  measure¬ 
ment  methodology  are  accomplished  considerable  benefit  to  both  psychological 
applications  and  to  behavioral  research  are  to  be  expected* 
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TAELS  xm 

Skill  attention  Tost  Mi&aicn  Switching  Iter-ora 
%  Phase  m&  ¥©rk*p&e® 


Crew/ 
Position/ 
Pilot _ 


C-4/P  SW 
1 
2 
3 

E  SW 
1 
2 
3 

C-9/P  SW 
1 
2 
3 

E  SW 
1 
2 
3 

O-8/F  SW 
1 
2 
3 

E  SW 
1 
2 
3 

0-13/p  SW 


PSrnae  # 


TEN/ 

cm  LOl 


BH  LPA  CA 


142 

21 

17f 

222 

22 

57 

25 

— — 

39 

9 

NP 

0 

2 

NP 

l(l9)** 

Iff 

2 

1 

NP 

4 

(24) 

2 

2 

258 

6f 

4 

Iff 

10 

111 

110 

17 

3 

3 

3 

3 

NP 

NP 

1(12) 

{SI 

1 

2 

3 

[114 

,118] 

1 

143 

21 

17t 

222 

22 

57 

25 

39 

HP 

1 

IIP 

1 

Iff 

1 

1 

NP 

(19) 

5 

(24) 

2 

260 

6f 

6 

0 

I6t 

112 

117 

:?t  1 

8 

1 

7 

5 

1  | 

Iff 

1 

(11) 

(111) 

25 

.108 

UP 

(11) 

6(111) 

6 

[114] 

1 

130 

24 

19 

27 

20 

24 

7 

10 

15 

2 

4  39 

2 

1 

1 

HP 

Iff 

NP 

NP 

223 

10 

20 

160 

16 

58 

47 

26 

98 

13 

21  2d 

3 

NP 

Iff 

1 

(44) 

1 

(20) 

Iff 

NP 

1 

1 

130 

24 

19 

27 

20 

24 

10 

15 

17 

6 

5  39 

NP 

Iff 

NP 

NP 

220 

10 

20 

160 

16 

58 

44 

21 

96 

9 

20  24 

1 

*> 

4 

NP 

Iff 

(15) 

1 

1 

NP 

NP 

i 

1 

! 

*  Lunar  Landing  and  Lunar  Ascent  data  were  broken  into  lesser  elements  only  for 
0-8  and  0-13.  CL  is  Coast  Descent  and  CA  is  Coast  Ascent, 
v  in  EA  of  0-4  the  Navigator  (P»3)  had  20  switches.  1  error j  0~9  (P”3)  20  switches, 
1  error?  0-8  (P-2)  53  switohes,  no  errors?  0-13  (P-2)  52  switches?  1  error. 

In  TIJ  for  0-4  and  0-9  the  navigator  had  4  switches.  Pilot  performance  order 
waa  P-2,  P-3,  P-1.  P-2  of  0-9  missed  1  switch. 
t  the  order  of  performance  in  these  instances  vac  P-3,  P-3,  P-2. 

**  parenthetical  numbers  indicate  switches  expected  when  different  from  ehove, 

NP  not  performed  by  this  Pilot. 
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APPENDIX  II 

Notes  on  Analytic  Methodology 


i?p ®m  n 


NOTES  OF  ANALYTIC  MgPMQEGLQST 


la  preparing  the  main  text  of  this  report  th©  intention  was  to  present 
enough  general  information  cm  th©  analytio  methods  used  to  sake  the  results 
stssaningful  and  interpretabla*  It  did  not  seas  ■wise  to  sake  fully  explicit 
all  the  initial  computations,  such  as  sight  ha  of  interest  to  one  wishing  to 
verify  a  portion  of  the  results  or  to  adopt  ths  aatho&s  to  his  om  nsadn.  Yet, 
being  quite  different  from  traditional  methods  in  orientation  and  intent,  the 
methods  used  should  be  fully  described.  Consaquently,  this  appendix  provides 
additional  step-by-step  details  on  the  computations  involved  in  obtaining  the 
basic  analytic  results,  from  which  the  results  described  in  the  tart  wore 
computed.  Mors  general  discussion  of  th®  underlying  measurement  philosophy 
may  be  found  in  a  separate  report  by  Co'cterman  (1967). 


As  already  described  (see  pages  24-27)  the  analytic  problem  began  with 
a  voluminous  collection  of  flight  control  data  for  each  of  12  pilots.  By 
careful  elimination  this  volume  was  reduced  to  sets  of  data  on  22  parameters 
scattered  throughout  the  simulated  lunar  landing  mission,  and  further  categor¬ 
ised  by  association  with  one  of  a  number  of  phases.  Most  of  these  22  measures 
represented  direct  records  of  flight  control  information,  although  four  of 
them  represented  simple  combinations  of  two  measures  (see  page  25)  which  were 
treated  in  the  same  fashion  as  the  raw  data.  Furthermore,  as  depicted  in 
appendix  IV,  for  each  pilot  with  respect  to  each  phase  there  were  records  of 
performance  on  first  a  variable  number  of  training  trials,  then  a  real  time 
mission,  then  the  fast  time  skill  retention  test  mission,  and  finally  on  a 
variable  number  of  retraining  trials  on  certain  mission  phases. 


Computation  of  Parameter  Probabilities  h>l 

The  first  step  in  interpreting  this  large  collection  of  data  was  to 
arrive  at  a  probabilistic  indication  of  each  pilot* s  performance  capability 
in  each  measured  aspect  in  training,  in  the  skill  retention  test  mission,  and 
in  retraining.  The  real  time  mission  was  omitted  from  consideration  because 
a  presumably  more  adequate  indication  of  pkill  achievement  was  available  from 
the  multiple  performances  just  prior  to  it  at  the  end  of  training.  Capability 
was  assessed  in  two  ways — first  with  reference  to  hypothetical  system  criteria 
for  the  measured  aspects  and  second  with  reference  to  capability  demonstrated 
by  the  individual  in  training  (his  p^  level  per  measured  aspeot). 

Reliability  par  oriteria  in  training.  Calculation  01  performance  capabil¬ 
ity  or  reliability  at  th®  end  of  training  with  regard  to  tho  hypothetical 
criteria  involved  three  simple,  though  tedious,  steps.  First,  having  ohosen 
to  consider  the  last  four  training  trials  (with  the  few  exceptions  noted  in 
the  text)  as  best  representing  that  capability,  means  and  estimated  standard 
deviations  ware  computed  from  each  sample.  The  formulas  used  are  the  con¬ 
ventional  ones  which  may  be  given  as  follows j 
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(2)  Estimated  StazuS&rd  Buriat  ion 


in  which  X  is  the  recorded  datum  and  n  is  the  mister  of  such  data  items 
(4  in  this  case). 

Bart,  on  the  assumption  that  re pasted  such  sets  of  four  performance® 
under  exactly  the  sas®  circumstances  would  te  found  to  follow  the  norml 
probability  density  function  ths  relevant  criteria  ware  interpreted  in  teams 
of  a  norml  function  having  those  respective  means  and  estimated  standard 
deviations.  Thus*  by  the  familiar  operation  of  dividing  the  difference  be¬ 
tween  the  mean  and  the  criterion  by  th©  estimated  standard  deviation  the 
distance  (j),  in  standard  deviation  unite,  of  the  criterion  from  the  ample 
mean  was  obtained  in  each  instance.  This  relationship  may  te  summarised  ass 

(3)  Standard  Deviations  from  the  Mean  *  ss  «*  X  -  C 

8 

in  which  C  is  the  criterion  for  a  particular  measured  aspect.  (See  table  F 
on  page  29  for  a  listing  of  the  hypothetical  system  criteria.) 

Th®  &  values  so  obtained  also  represent  the  number  of  standard  devia¬ 
tions  of  the  criterion  fro®  the  mean  of  a  norml  distribution  having  a  moan 
of  zero  and  a  standard  deviation  of  one,  for  which  the  proportions  above  and 
below  a  particular  deviation  are  well  known.  Accordingly,  in  the  final  ®top, 
the  j£~value  obtained  for  each  measured  aspect  and  each  pilot  was  interpreted 
as  a  probability  by  simple  interpolation  fro®  a  standard  table  of  the  norml 
probability  integral  (Peters  and  Fan  Voorhis,  1940,  pages  481-4)*  The 
probability  taken  ma  always  the  proportion  of  the  hypothetical  function 
(based  on  the  sample  statistics)  indicating  performance  superior  to  the 
criterion.  In  actuality,  this  was  performance  numerically  sssller  than  the 
criterion. 

Thus,  for  example,  to  determine  the  capability  of  P-131  to  meet  the 
criterion  of  a  10  fps  rate  at  impact  on  lunar  lending  the  recorded  rates  on 
the  last  four  training  trials  were  taken  as  the  sample.  These  rates  ware 
2.8,  5.2,  5»6  and  5*8  ffcs.  The  I  of  these  values  was  found  to  te  4.85  and 
the  was  found  to  te  I.3892,  as  computed  by  formulas  1  and  2.  Also,  by 
formula  3,  the  difference  of  5*15  between  criterion  and  Dean  was  found  to 
give  a  ^  of  3.7072.  Or,  described  in  another  way,  the  mean  of  the  sample  was 
found  to  be  nearly  four  standard  deviations  superior  to  (though  numerically 
less  than)  criterion  performance.  According  to  the  tabled  values  of  j.  for 
the  normal  probability  integral,  somewhat  fewer  than  5  oases  in  10,000  would 
be  erpected  to  fall  beyond  a  &  of  3.7072,  and  thus  to  axoeod  the  criterion, 
whereas  the  remaining  9>995  or  more  cases  would  te  expected  to  te  less  than 
the  criterion.  Therefore,  the  probability  of  P-131  meeting  the  criterion 
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of  a  10  fps  rate  at  impact  on  lunar  landing  is  >.999»  as  given  in  table  7T7 
(appendix  III).  This  circumstance  is  Illustrated  in  figure  10,  in  Kfaici,  th© 
hypothetical  distribution  of  ?“131’a  performance  at  the  end  of  training  is 
shown  with  relevant-  values  noted. 

Reliability  per  criteria  in  test.  The  assessment  of  capability  in  th© 
skill  retention  test  followed  the  earn®  logic,  but  with  variations  in  calcula¬ 
tion  necessitated  by  the  availability  of  only  one  measurement.  If  the  test 
of  each  pilot's  skill  had  involved  multiple  performances  then  it  would  have 
been  possible  to  take  some  (or  all)  of  them  as  a  sample  and  to  proceed 
exactly  as  with  the  training  data  to  arrive  at  the  likelihood  of  success  in 
meeting  each  criterion.  However,  in  order  to  avoid  inaccuracy  because  of 
relearning  only  the  first  performance  after  th©  retention  period  could  be 
considered  and,  with  only  one  measurement,  neither  the  mean  nor  the  estimated 
standard  deviation  could  be  computed. 


Figure  10.  Hypothetical  Distribution  at  the  End  of  Training  of  P-131's 

Vertical  Rate  at  Impact  on  Lunar  Landing  indicating  Probability 
of  Meeting  the  Criterion 
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This  analytic  problem  was  resolved  by  assuming  that  the  obtained  value 
constituted  the  best  available  estimate  of  the  msan  and  that  the  variability 
in  performance  at  the  end  of  training  is  indicative  of  i&at  mild  be  observed 
in  multiple  test  performances  under  th©  seme  circumstances  (if  they  war© 
possible).  That  variability  is  to  be  orpaerted  implies  that  the  single  perform¬ 
ance  would  not  be  the  earns  as  an  observed  mean.  Also,  if  performance  change 
between  training  and  test  is  to  be  expected  then  th©  variability  in  performance 
may  be  expected  to  change  in  some  degree.  Consequently,  acme  inaccuracies  are 
to  b®  expected  in  the  use  of  these  as gumptions,  nevertheless,  the  estimates 
of  capability  so  obtained  are  based  on  tha  best  available  information  and  are 
probably  biased  as  a  group  to  only  a  slight  extent,  as  discussed  on  page  ?6. 

Implementation  of  these  assumptions  in  calculating  the  estimated 
probabilities  was  comparatively  simple.  With  th©  test  performance  (X^)  and 
a.  for  training  already  available,  to  arrive  at  the  standard  deviations  between 
test  and  criterion  performance  (j^)  it  was  merely  nacsRsacy  to  divide  the 
actual  difference  between  them  by  the  borrowed  standard  deviation  estimate, 
as  in  the  following! 

(4)  zM  =  ImZSl 


in  which  is  the  recorded  mission  performance  and  3^  is  the  estimated 
standard  deviation  of  performance  in  training  on  a  particular  measured  aspect, 
Again,  the  proportion  of  values  expected  to  be  numerically  less  (hence  superior) 
to  a  jj,  aa  great  as  that  obtained  was  taken  as  the  probability  of  successfully 
meeting  the  criterion. 

By  way  of  further  illustration,  P-131'b  lunar  landing  impact  rate  of 
5.4  fpB  in  the  test  mission  is  4*6  fps  less  than  (or  superior  to)  the  criterion. 
This  difference  is  3.3113  times  the  standard  deviation  of  1,3892  Spa  found  in 
training-  suggesting  that  slightly  more  than  9,995  such  test  performances  in 
10,000  distributed  normally  with  a  mean  of  5*4  fps  and  a  standard  deviation  of 
1,3892  fps  would  not  exceed  the  criterion  of  10  fps.  Consequently,  P-131's 
reliability  in  controlling  the  lunar  landing  impact  rate  in  the  retention  test 
waa  considered  to  be  >.999,  as  shown  in  table  XXVI  (appendix  III). 

Reliability  _Pbr„crLt.9rda  The  assessment  of  capability 

to  meet  criteria  in  retraining  required  new  calculations  but,  unlike  that  _or 
test  performance,  presented  no  complications  because  multiple  measures  were 
available.  Thus,  in  general,  reliabilities  per  criteria  in  retraining  were 
computed  in  exactly  the  same  way  as  reliabilities  per  criteria  at  the  end  of 
training.  A  sample  of  performances  was  identified  and  X  and  s,  computed  from 
that  sample.  Then,  g;  with  respect  to  the  criterion  was  calculated  and  inter¬ 
preted  as  the  probability  (p)  of  successful  performance.  Of  course,  as 
described  in  the  text,  for  the  six  pilots  comprising  two  crews  (0-8  and  0-13) 
this  was  done  both  with  reference  to  the  first  four  and  the  best  four  trials 
of  retraining  (taken  in  successive  four-trial  blocks).  Calculation  of  the 
reliabilities  of  the  six  pilots  comprising  C-4  and  0-9  who  were  tested  earlier 
was  based  upon  all  their  retraining  trials  aa  a  sample  because  they  hail  fewer 
trials.  But  these  are  differences  in  sampling,  not  in  method  of  calculation. 
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The  second  sasese&snt  of  capability,  rather  then  referring  to  tha  hypothetical 
cyst  ess  criteria,  referred  to  a  level  of  perfersaaoe  in  each  se&sured  aspect 
which  each  Individual  had  deaoastmted  capability  to  achieve  with  hig£  reli¬ 
ability,  Instead  of  an  arbitrary  and  agitate  criterion  ©omon  for  the 
analysis  of  all  the  pilots’  performances,  in  this  assessment  the  rafsranoe 
was  to  an  indivldually^estahlished  level  of  performance  varying  with  the 
individual  and  his  own  former  skill,  The  value  of  this  second  assessment 
was  its  greater  sensitivity  to  ahif is  in  performance  in  the  test  mission  and 
in  retraining,  from  that  observed  at  the  end  of  training. 

Calculation  of  these  probabilities  for  each  pilot  on  each  ue&srured 
aspect  involved  one  new  calculation  and  a  staple  manipulation  of  the  result 
in  conjunction  with  certain  values  already  available  fro®  the  analysis  with 
respect  to  criteria.  Earing  chosen  a  probability  of  *950  a a  a  reasonable 
compromise  between  certainty  of  achievement  and  estimation  error*,  the  first 
step  was  to  estimate  for  each  measure  and  pilot  the  level  of  performance  to 
be  expected  in  950  performances  of  1,000  at  ths  end  of  training  (designated 
X  qr).  This  was  accomplished  by  reference  ones  again  to  the  normal  distribu¬ 
tion  which  is  known  to  include  95^  of  the  oases  from  one  extreme  to  1,6449 
standard  deviations  in  the  direction  of  the  other  extreme.  Thus,  bo  find  the 
X  (-c  reference  it  was  saersly  necessary  to  add  1,6449  standard  deviations  to 
tfir  mean,  as  in  the  following: 

(5)  *  X  +  1.6449.? 


Subtraction  instead  of  addition  would  have  been  required  if  the  measures  had 
increased  rather  than  decreased  with  superiority  in  performance.  The  X  „  so 
obtained  provided  the  reference  performance  level,  analogous  t-o  a  systdlr  5 
criterion,  to  be  used  in  this  second  assessment  of  capability  in  teat  and 
retraining. 

With  the  X  qr  available  for  each  measured  aspect  and  pilot  it  was  then 
a  simple,  rapetifcl&is  operation  to  estimate  the  probabilities  of  achievement. 
As  in  the  analysis  with  regard  to  criteria,  the  difference  between  mean 
performance  (or  test  performance  taken  as  the  mean)  and  the  X  «c  divided  by 
the  estimated  standard  deviation  indicates  the  number  of  stanSard  deviations 
of  the  Xo95  from  the  mean.  For  the  test  performance  this  relationship  is 

* 

(6)  2m/.9S  =  ■^•■35 

ST 

in  whioh,  as  in  formula  4,  the  computed  from  training  performance  is  used 
as  the  estimated  variability.  For  retraining  performance  the  relationship  is 

(7)  ZR/.*S  ""  Xu  ~  X,9i 

Sk 


in  which  the  subscript  B  indicates  that  the  value  was  computed  from  the 
retraining  E-aesple,  Both  expressions  (6  and  7}  are  the  same  as  thair  counter- 
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parts  for  ocwput ing  £  with  respect  to  criteria  except  bha.  X  is  substituted 
for  C.  The  resulting  ^-values  are  likewise  used  to  obtain,  &y  interpolation 
from  a  table  of  the  normal  probability  integral,  the  expected^  presort  ion  of 
cases  superior  to  (or  less  than)  the  r&fersnca — in  this  case  X 


Thus,  pursuing1  the  previous  illustrative  data  still  further,  P~131'b 
impact  rates  on  his  last  four  lunar  landings  in  training  suggest  (by  applica¬ 
tion  of  formula  5). that  in  95  out  of  100  performances  his  rate  would  not  exceed 
7.1351  fps.  ‘This  £05  differs  by  1.7351  fps  or  1.249  estimated  standard  devia¬ 
tions  (of  1.3892  fps)  from  hia  retention  teat  rate  of  5«4  fpa.  k  deviation 
that  great  at  one  extreme  of  the  normal  distribution  is  expected  to  occur  only 
about  106  times  ir.  1,000.  Consequently,  the  likelihood  in  the  test  mission 
of  P-131  meeting  his  former  p^c  level  of  7*1351  ?ps  i»  rate  at  impact  on 
lunar  landing  is  taken  fco  be  This  outcome  indicates  a  slight  103s  from 

training  to  test  in  capability  to  control  impact  rate  (i.e.,  a  decrease  in 
likelihood  of  from  the  reference  of  .950). 


Furthermore ,  in  his  best  four-w.....!  v-,->ok  of  retraining  P-13 I* s  impact 
rates  were  3.0,  1.4,  3.4,  and  2.6  fps  with  a  i  u.  ^  ^  «r»d  *n  Sr,  of  ,864  fpa. 
The  X  05  of  7.1351  computed  from  the  training  data  is  found  to  "differ  by  5*249 
estimated  standard  deviations  from  the  of  2.6  and  a  deviation  that  great 
in  the  same  direction  is  expected  to  occur  considerably  fewer  than  five  times 
in  10,000  performances.  The  estimated  likelihood  in  retraining  of  P-131 
successfully  realising  his  former  p  level  of  impact  rate  control  on  lunar 
landing  is,  then,  in  excess  of  .9995»^  This  represents  a  considerable  increase 
in  capability  over  that  demonstrated  at  the  end  0?  training. 

SfflEa&ar-  o£ .JgSSS&ESg. . j£Z  obtaining  parameter  probabilities  (p).  By  the 
use  of  the  simple  methods  just  described  and  illustrated  all  the  values  tabled 
in  appendix  III  were  obtained.  These  values  provide  tho  desired  probabilistic 
indication  of  each  pilot *s  performance  capability  in  each  measured  aspect  of 
training,  test  mission,  and  retraining  to  serve  as  a  basis  for  assessing, 
more  inclusively,  phase,  mission,  and  crew  capabilities.  None  of  the  computa¬ 
tions  required  are  unusual  or  complicated,  but  whsn> there  are  many  of  them  and 
they  are  accomplished  manually  (as  in  this  instance)  they  do  require  consider¬ 
able  time.  Therefore,  whan  applied  to  data  of  much  volume  these  methods  would 
be  most  efficiently  carried  out  ’  7  moans  of  computer  analysis.  The  following 
summary  of  procedural  steps  required  in  this  initial  analysis  suggests  the 
ease  with  which  the  analysis  might  have  been  programmed  for  digital  computa¬ 
tion. 


1.  Identify  and  note  (enter)  all  reference  performance  consisting  of i 

a.  hypothetical  system  criteria  (C) 

b.  trainirg  performance  sample  (X<p) 

c.  test  mission  performance  (Xjj) 

d.  retraining  performance  samples  (Xg) 
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parts  for  computing  £  with  respect  to  criteria  except  that  X  qk  is  substituted 
for  C.  The  resulting  gr-valuea  are  likewise  used  to  obtain,  ^interpolation 
from  a  table  of  the  normal  probability  integral,  the  expected  proportion  of 
cases  superior  to  (or  less  than)  the  reference — in  this  case 

Thus,  pursuing  the  previous  illustrative  data  still  further.  P-131 *s 
impact  rates  on  his  last  four  lunar  landings  in  training  suggest  (by  applica¬ 
tion  of  formula  5 5. that  in  %  out  of  100  performances  his  rate  would  not  exceed 
7.1351  fps.  This  X.o5  differs  by  1.7351  fps  or  1.249  estimated  standard  devia¬ 
tions  (of  1.3892  fps]  from  his  retention  test  rate  of  5*4  fps*  A  deviation 
that  groat  at  one  extreme  of  the  normal  distribution  is  expected  to  occur  only 
about  106  times  in  1,000.  Consequently,  the  likelihood  in  the  test  mission 
of  P-131  meeting  his  former  p  os  level  of  7.1351  fps  in  rate  at  impact  on 
lunar  landing  is  taken  to  be  loy4*  This  outcome  indicates  a  slight  loss  from 
training  to  tast  in  capability  to  control  impact  rate  (i.e. ,  a  decrease  in 
likelihood  of  .05 6  from  the  reference  of  .950). 

Furthermore,  in  his  best  four-trial  block  of  retraining  P-131’ s  impact 
rate?  were  3.0,  1.4*  3.4*  and  2.6  fps  with  a  Xg  of  2,6  and  an  s«  of  .864  fps. 
The  £05  of  7.1351  computed  from  the  training  cats  is  found  to  differ  by  5*249 
estimated  standard  deviations  from  the  X%  of  2.6  and  a  deviation  that  great 
in  the  same  direction  is  expected  to  occur  considerably  fewer  than  five  times 
in  10, OCX)  performances.  The  estimated  likelihood  in  retraining  of  P-131 
successfully  realizing  his  former  p  nr  level  of  impact  rate  control  on  lunar 
landing  is,  then,  in  excess  of  .9995*  This  represents  a  considerable  increase 
in  capability  over  that  demonstrated  at  the  end  of  training. 

%  the 

use  of  the  simple  methods  Just  described  and  illustrated  all  the  values  tabled 
in  appendix  HI  war®  obtained.  These  values  provide  the  desired  probabilistic 
indication  of  each  pilot’s  performance  capability  in  each  measured  aspect  of 
training,  test  mission,  and  retraining  to  serve  as  a  basis  for  assessing, 
more  inclusively,  phase,  mission,  and  crew  capabilities.  None  of  the  computa¬ 
tions  required  are  unusual  or  complicated,  but  when  there  are  many  of  them  and 
they  are  accomplished  manually  (as  in  this  instance)  they  do  require  consider¬ 
able  time.  Therefore,  when  applied  to  data  of  much  volume  these  methods  would 
be  most  efficiently  carried  out  by  means  of  computer  analysis.  The  following 
summary  of  procedural  steps  required  in  this  initial  analysis  suggests  the 
ease  with  which  the  analysis  might  have  been  programmed  for  digital  computa¬ 
tion. 

1.  Identify  Mad  note  (enter)  all  reference  performance  consisting  oft 

a.  hypothetical  system  criteria  (C) 

b.  training  performance  sample  (Xj) 

o.  test  mission  performance  ( X 

d.  retraining  performance  samples  (Xg) 
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2.  Compute  ?,  jb>  and  for 

ae  training  performance  sample 

b.  retraining  performance  samples  (t  not  used) 

3.  Compute  all  ^  for 

a,  training  as  ztH  --  IOLt  *“  C 

>?T 

b0  test  mission  as  Zc/m  -  C 
c»  retraining  as  zc/R  =  %r  ~  C 

4.  Compute  all  jg^,.  for 

» 

a.  test  mission  as  Z9S/m  =  ~ 

S? 

b.  retraining  as  Z.es/a  "  %r  ~  ^.es/r 

5r 

5.  Interpret  all  &  as  single-tailed  probabilities  (p) 


With  the  probabilities  describing  each  pilot's  performance  capability 
in  each  measured  aspect  of  training,  test  mission,  and  retraining  available 
it  ms  a  simple  matter  to  obtain  the  various  mors  inclusive  probabilities 
reported  in  the  fashion  described  on  pages  27-28  and  33.  Only  two  types 
of  calculations  were  involved— although  some  of  the  indicants  obtained  (for 
example,  ?)  required  both  kinds.  In  the  one  operation,  the  results  of  tdxich 
are  always  designated  by  a  bar,  the  sum  of  the  values  was  simply  divided  by 
their  number  to  obtain  an  average  or  cumulative  probability  for  the  set.  In 
the  other  operation,  the  results  of  which  are  always  designated  by  a  dot,  the 
joint  product  of  the  values  was  taken.  The  order  in  which  these  calculations 
were  performed  to  obtain  any  given  indicant  is  always  shown  by  the  upward 
progression  of  these  bar  and  dot  superscripts  from  the  basic  symbol.  For 
example,  in  obtaining  i>,  the  joint  produots  of  the  separate  probabilities 
within  phases  were  calculated  first  and  then  these  produots  were  taken 
cumulatively.  With  an  awareness  of  these  coding  procedures  all  of  the 
probabilities  reported  in  the  text  may  be  interpreted  readily. 


Reliability  Per  Criteria  at  the  End  of  Training  (Last  Pour  Triala) 
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For  C q  and  C-^  based  on  the  first  four  training  trials,  for  and  based  on  all 

retraining  data  (generally  2  to  8  trials,  depending  on  phase  as  noted  in  parentheses) 
t  Dash  indicates  value  exceeding  .9995 
#  Based  on  reduced  data 
ND  Phase/parameter  not  dene 
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Graphs  of  "Data  from  P-131 
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